\platforms: AI-Enhanced Interactive Narratives for Programming Education

Researchers and practitioners have recognized the value of Artificial Intelligence (AI) as a learning tool (heaven2023chatgpt, ; hirsh2023chatgpt, ), particularly for Self-Directed Learning (SDL) (lin2023exploring, ). Prior work has integrated AI and dialog systems into education in a variety of ways. Conversational agents and chatbots have been used to support personalized learning experiences through natural dialog (han2021designing, ; li2022chatbot, ; luo2020code, ). Intelligent tutoring systems can adaptively provide hints and feedback to learners, similar to human tutors (jell2023towards, ). They can also be particularly useful for learning new natural languages (bibauw2022dialogue, ). These systems demonstrate the potential for AI to enable responsive and adaptive education.

More recent work on AI in education has also examined Large Language Models (LLMs) in particular. LLMs can answer student questions (hirsh2023chatgpt, ; joyner2023chatgpt, ; liffiton2023codehelp, ), generate explanations (leinonen2023explanations, ), and even solve many coding problems (kazemitabaar2023studying, ; reeves2023evaluating, ). However, if not constrained properly, LLMs may produce misleading, incorrect, or educationally unproductive responses (guo2023six, ). Effectively aligning LLMs interactions with learning goals remains an open challenge.

Within this context, platforms like Graphologue (jiang2023graphologue, ) and Sensecape (suh2023sensecape, ) both explore ways to leverage LLMs to arrange learning materials. However, Graphologue’s design is oriented towards building a deep understanding of one particular concept by exploring multiple levels of abstraction for LLM responses and Sensecape’s design is oriented towards sensemaking. Neither is focused on longer educational narratives, as the EDBook platform is. The features of Graphologue could be combined with the EDBook platform to produce narratives that have more ‘local’ detail throughout.

One feature of the EDBook platform is that it can represent and help authors automatically generate dialogic content between any number of agents, based on descriptions of those agents. Commercial tools like character.ai (CharacterAI, ) also allow users to create conversations with simulated agents. Similarly, Markel et al. explored how LLMs can simulate students to help train instructors (markel2023gpteach, ). Unlike these platforms, EDBook dialogs are designed to guide readers towards targeted learning goals.

Since modern LLMs like GPT-3 have proven to be capable of solving a variety of problems (bubeck2023sparks, )—particularly for programming problems—researchers have recognized that LLMs are likely to have a profound effect on almost every aspect of programming education (becker2023programming, ). This includes how we assess understanding, how we generate practice exercises (lu2023readingquizmaker, ), how we create programming tutorials (guo2023six, ), and what is pedagogically important to teach (e.g., perhaps de-emphasizing low-level implementation work that can be automated). Prior work has studied many aspects of the impact of LLMs on programming education. Kazemitabaar et al. found that code generation tools can be a valuable learning tool in introductory programming education and that using code generation tools in the context of learning could reduce frustration without negatively impacting students’ understanding (kazemitabaar2023studying, ). Sarsa et al. found that LLMs can also generate high-quality coding exercises that can be directly used by instructors in their courses (sarsa2022exercise, ). Additionally, the code explanations produced by LLMs also demonstrate a high level of correctness, making it a promising tool with the potential to expedite tutoring processes for teaching assistants (leinonen2023explanations, ; sarsa2022exercise, ).

For novice programming students, the incorporation of LLMs could potentially reframe their educational objectives. There arises a heightened emphasis on acquiring the proficiency to explain and communicate algorithms, as it directly influences the accuracy of code produced by LLMs (yujia2022science, ). In addition, the significance of code debugging and extending should also be emphasized, given that LLMs is capable of generating basic code for learners (becker2023programming, ). By leveraging the capabilities of LLMs, students can more easily understand error messages, identify mistakes, and accelerate their overall learning progress (leinonen2023error, ).

The incorporation of LLMs into programming education also raises certain concerns. Code generated by LLMs can exhibit biased structures and comments (pearce2022unsecure, ) and may also contain insecure vulnerabilities (chen2021evaluating, ). Furthermore, Becker et al. suggest that the coding style and approaches adopted by LLMs might prove too advanced for novice programmers to readily adopt (becker2023programming, ). In this case, EDBook aims to create a semi-structured learning environment, enabling learners to engage with LLMs under appropriate, dialogue-style guidance. This approach is designed to mitigate the potential negative impacts that could be transferred to learners.

There are several interactive textbook platforms, particularly for teaching programming. Runestone (ericson2020runestone, ), a platform on which several widely-used programming textbooks are built, enables interactive content, including code-writing questions, multiple choice, and Parsons problems (denny2008evaluating, ) (where students drag and drop code blocks to arrange them in a correct order rather than writing code manually). Jupyter Book (executable_books_community_2020_4539666, ) is another computational platform that empowers the creation of interactive textbooks. It integrates narrative text, code execution, visualizations, and interactive widgets within the environment. Al-Gahmi et al. emphasize the effectiveness of using Jupyter as a valuable tool in computing education within the classroom, leading to significant improvements in students’ performances on assignments (al-gahmi2022jupyter, ). OpenDSA (shaffer2011opendsa, ) is an open-source platform designed to streamline the creation of interactive textbooks for computer science-related subjects. It enables instructors to create visualizations, quizzes, and animations within the web-based book, enhancing the learning experience for students. Prior studies indicate that many students prefer interactive textbooks over conventional text materials (smith2021modeling, ; tommy2016opendsa, ). However, existing interactive textbook solutions have certain limitations. Computational notebooks such as Jupyter Notebook might not be intuitive for novice learners (johson2020jupyterinclassroom, ). On the other hand, other web-based textbooks can lack support for complex code execution and note-taking (miller2012beyondpdf, ). Consequently, students need to use multiple platforms simultaneously. The EDBook platform is designed as an extension of the Visual Studio Code platform. By integrating learning and coding within a single interface, EDBook aims to help students engage with content and practice coding cohesively.

Goal-oriented dialog systems are dialog systems that give users agency and choice while still directing them towards a set goal. Goal-oriented dialogue systems are commonly applied to chatbots (grudin2019chatbots, ; xie2022converse, ), which are developed to provide real-time assistance to users in accomplishing specific tasks. Prior studies indicate that numerous goal-oriented dialogue chatbots struggle to meet users’ expectations (jain2018evaluating, ; zamora2017m, ). This challenge arises due to the intricacies of automating certain tasks (cranshaw2017calendar, ), the complexities of dialogues (grudin2019chatbots, ), and the inherent difficulty in accurately identifying natural language content (li2020conversation, ). Most systems are targeted towards goals that are specific and concrete, such as booking a flight or scheduling a meeting (muise2019planning, ; santos2022review, ). By contrast, notebooks in EDBook have the goal of building conceptual understanding for readers. To mitigate the challenges presented by more open-ended goal-oriented dialog systems, EDBook adopts a tree-based scripted dialogue style for its notebooks to minimize dialogue variability. Additionally, the preset message options can also reduce potential interruptions during dialogues (li2020conversation, ). In EDBook, the learning objectives can be considered “abstract tasks”. To aid readers in identifying their “task progress”, EDBook incorporates quizzes within its notebooks. Many tools, including Dialogflow (Dialogflow, ) and Tae et al.’s work (kim2023cells, ), use state-based representations of dialogs. This feature supports users to go back to the previous chat boxes and modify any cell to their favor. Extensive research has also been done in predicting future utterances from textual data, such as example-based dialog systems (lee2006situation, ). Example-based dialog systems can generate system responses for user input based on stored data examples.

After we have considered possible visualized dialog representations from traditional spoken dialogs (dahl2005visualization, ), we finally decide to use representations of dialogs that are both state-based and tree-based. Notably, our work is designed and implemented as a tool to help users in navigating programming instructional content. It serves the purpose of guiding users through the proposed learning path while also alerting them if their progression diverges from the initially intended learning trajectory. This is achieved by a seamless amalgamation of the state-based and tree-based representations. Moreover, the system enables users to embark on exploratory journeys, empowering them to attempt different paths within the instructional dialogue. If users acquire new insights during their exploration and decide to revise their approach, the system facilitates a fluid return to previous conversations.

Topic shifting (tang2019target, )—shifting open-ended conversations towards a target topic—is also a promising approach. However, existing topic-shifting approaches are designed for conceptually simple topics (single entities, like ‘football’) (tang2019target, ) rather than complex learning goals. Further, topic shifting relies on LLM-generated content, which can be inaccurate or misaligned with learning goals.

chat.codes (oney2018creating, ) is a tool for discussing code. Although the use case of EDBook is different, there is overlap in the features of EDBook and chat.codes. Like chat.codes (oney2018creating, ), EDBook enables programmers to create deictic references between messages and specific regions of code. However, EDBook’s design of these deictic references differs from that of chat.codes in several important ways. First, deictic references in chat.codes needed to be specified on the character level through markdown annotations. For example: “[these lines](code.js:L23-L42)” creates a deictic reference in chat.codes. If the user hovered over “these lines”, lines 23–42 in code.js would be highlighted. In our experimentation with writing content, we found that we rarely needed to write fine-grained references. Thus, deictic references in EDBook are made at a cell level—cells contain a list of references, rather than specific parts of text. We found that this change simplified deictic pointers for both authors and readers. For authors, EDBook reduces the amount of syntax they need to learn and allows them to add deictic references through simpler point-and-click interactions³³3chat.codes includes User Interface (UI) functionality for adding deictic references without typing out the syntax manually. Users can highlight a section of an un-sent message and then highlight a region of code to add a reference to the highlighted part of the message. However, we found that this interaction was not discoverable and could be difficult for authors, particularly when they made subsequent modifications.. For readers, EDBook’s representation of deictic pointers allows them to see them automatically. Unlike in chat.codes, where readers needed to actively hover over the relevant text to see pointers, EDBook can always display deictic references for the selected cell(s). Further, EDBook visualizes a curved line between the selected cell(s) and the code they discuss, which helps form a clearer connection between the narrative and code.

Callisto (wang2020callisto, ) also contains deictic references, in the context of the Jupyter Notebook platform. Callisto’s deictic references are also coarser than those of chat.codes, with Like EDBook, Callisto’s references are made on a cell-level, where relevant code will be highlighted when one message is selected. The primary distinction between Callisto and EDBook lies in their approach to highlighting code for deictic references. In Callisto (wang2020callisto, ), a highlight color is used to emphasize relevant content, including not only code but also notebook cells, markers, snapshots, and different versions. This highlight color strategy maintains a uniform experience across various content types, effectively highlighting the contextual discussion of selected messages. In contrast, the EDBook utilizes a curved line to connect relevant code and cells. This method is designed primarily for code referencing, which creates a more intuitive connection between code and narrative cells.

Colaroid (wang2023colaroid, ), another extension for Visual Studio Code, also combines narrative text with iterative code changes for authoring coding tutorials. Like EDBook, Colaroid enables authors to explain larger pieces of code through a series of steps. However, there are several key differences between Colaroid and EDBook. First, Colaroid does not contain any dialogic features—readers cannot interrupt the narrative to query an LLM and authors cannot build assessments (e.g., multiple-choice or coding questions) into their narratives. Second, Colaroid does not include deictic references to better tie the narrative text with content. Finally, EDBook gives users more agency in choosing how to follow a given tutorial. Readers can select which response they want to give and can write their own parts of narratives. However, Colaroid also has several features that EDBook would benefit from—most notably, the ability to integrate with git and version control tools and the ability to re-play user interactions on a UI.

Like the EDBook platform, Torii (head2020composing, ) also allows code examples to be built up incrementally. However, beyond this similarity, there are significant differences between the design and use cases for these two tools. \platforms focus more on explanation, exploration, and learning through interactive dialog whereas Torii is designed to give readers clear walkthroughs of a single codebase.

The core design of the EDBook platform is in how it combines dialog trees with open-ended LLM interactions. The fundamental design questions are about balancing the ability for learners to explore while meeting concrete learning goals and how to represent dialogs in a way that is coherent and understandable.

EDBook also contains many features that complement its core interactions. These features are not novel individually but are crucial for the effectiveness of EDBook as a learning tool.

As far as we are aware, EDBook is the first platform to combine pre-written dialog trees with context-informed LLMs. This design mitigates unhelpful LLM responses and keeps learners focused while allowing flexibility. The dialog trees provide goal-oriented structure that we can guarantee to be accurate while the LLM queries enable personalized explorations. The technique is novel in using dialog state for education-oriented LLM guidance. Further, the design of EDBook’s deictic references (visually tying messages with the code being discussed) is unique relative to other tools that contain deictic code references (wang2020callisto, ; oney2018creating, ).

Finally, complementary capabilities (incremental code edits, deictic references, and interactive assessments) enrich the core dialogic learning in EDBook. They demonstrate how interactive features can be effectively integrated to engage learners. The combination enables more pedagogically productive programming narratives.

We reached out to students enrolled at (INSTITUTIONS REMOVED FOR ANONYMITY) through email. Participants were required to have taken at least one Python programming course but no prior experience with the specific content covered in the tutorials—no experience with Scheme or LISP (one task in our study) and no experience with writing code for caching web requests in Python (another task in our study). We determined eligibility by participants’ responses on a screening form.

Of the eligible participants, 20 completed the study. We selected participants based on their eligibility and aimed for a diverse set of backgrounds. One additional participant began the study but stopped the study after performing one task due to internet connectivity problems. All participants were students, as learners are the primary audience for \platforms. Most participants (16 of 20) were in degree programs that involved programming (either Computer Science or Information), as we required some prior technical knowledge. Relevant demographics of the 20 remaining participants are itemized below:

All participants had prior experience using ChatGPT and 15 of 20 had used ChatGPT in their own learning, primarily for programming support. All but one participant had used Visual Studio Code.

Our study had two conditions, illustrated in Figure 6:

• •

  Condition 1 ($\mathcal{C}$_EDBook): Participants read educational dialogic EDBook content.

• •

  Condition 2 ($\mathcal{C}$_Web+GPT): Participants read the same educational content but in non-dialogic format and translated into a standard webpage format. They also had access to ChatGPT, with the content provided as background context to ensure it produced responses that were consistent with the content.

And two tasks:

• •

  Task 1 ($\mathcal{T}$_scheme): In this task, participants learned the basics of the Scheme programming language (a functional language that is a dialect of Lisp). This task was designed to represent the challenges of learning content that is conceptually new, as most participants have no prior familiarity with any form of declarative programming.

• •

  Task 2 ($\mathcal{T}$_cache): In this task, participants walked through how to write Python code that makes web requests (using the requests module) and caches the result for a pre-specified period of time. Participants had familiarity with the underlying programming language (Python) but not with this specific application or algorithm. This task was designed to represent the challenges of learning through building. Although none of the underlying concepts were new to participants, the specific way they were put together in a larger (approximately 70-line) code sample could be difficult to follow.

In both conditions, tasks were split into smaller topic-focused chapters. These chapters included quiz questions and other methods to allow readers to check their understanding. These quiz questions were different from the assessment questions we used after each task to asses their understanding. In both conditions, participants interacted with OpenAI’s ChatGPT-3.5.

Figure 7 illustrates our study design. The study coordinator began each study by asking participants to complete a consent form and informing them of the study procedure. They then completed two learning tasks ($\mathcal{T}$_scheme and $\mathcal{T}$_cache) in both conditions ($\mathcal{C}$_EDBook and $\mathcal{C}$_Web+GPT), in a latin square design to counter order effects. Before each task, participants spent approximately 10 minutes walking through the interface(s) they would interact with for that condition (instructions on how to use \platforms or ChatGPT). After each task, participants completed assessments to gauge their understanding of the materials. They were given 10 minutes for each assessment and any incomplete answers were marked as “incorrect”. During these assessments, they were allowed to go back and check the learning materials but they were not allowed to send further queries to the LLM (to prevent participants from feeding assessment questions back to the LLM without demonstrating an understanding of the material). We also asked participants to fill out a short survey after each task. After participants completed both tasks, the study coordinator conducted a post-task interview.

All study sessions were conducted virtually. With participants’ approval, the study coordinators recorded their screens and collected other usage telemetry information for subsequent analyses. Each session took approximately two hours and participants received $50 (USD) as compensation. Our institution’s Institutional Review Board (IRB) approved our study setup.

We describe our quantitative results below and summarize our interview responses and findings in section 4.5.

Based on the results described above and interviews with participants, we identified several key findings.

Our results provide important insight into how participants interact with educational content (both \platforms and “standard” content with ChatGPT). However, as with any controlled study, there are important limitations to consider with respect to the results. First, our participant pool is not representative of the demographics of all learners. Second, participant feedback to EDBook content might have been more positive for participants who wanted to play the role of a “good participant”. Finally, our study was a short term study and is not necessarily representative of longer-term usage patterns.

Still, we believe our results are indicative of realistic interaction patterns. Overall, we found that participants appreciated many of the interactive parts of \platforms and found them more engaging. In our study, this increased engagement did not lead to improved assessment scores (assessment score averages in $\mathcal{C}$_EDBook were higher and time spent on assessment in $\mathcal{C}$_EDBook was lower but neither difference with $\mathcal{C}$_Web+GPT was statistically significant). However, many prior studies have found that increased learner engagement improves learning outcomes (gray2016effects, ).

Participants also pointed to ways that the EDBook platform could improve: (1) improving navigability and (2) promoting critical thinking. We believe that navigability (1) could be improved with relatively small implementation changes—writing more concise dialogic content and engineering better “search” features. However, promoting critical thinking (2) might require more substantial design changes, as we will discuss in section 6.3. Some of the challenges participants faced in $\mathcal{C}$_Web+GPT could also be addressed with design changes. For example, several browser extensions that integrate LLMs with content (for example, by clicking a portion of content to provide it as context to an inline chat window) could make it easier to provide local context.

Figure 8 illustrates the structure of EDBook content files (JSON data with a .dpage extension). The format of EDBook content was inspired by that of Jupyter’s .ipynb Notebooks (kluyver2016jupyter, ). However, some notable differences with Jupyter’s format are (1) a tree structure for cells (rather than lists); (2) a notion of users and provenance to track who sent which message; (3) “pointers” that better link narratives with code; (4) code diffs that allow code to change as the narrative advances; (5) education-specific “directives” such as multiple-choice questions and coding questions; and (6) separating users’ state from the page data. Every node in the dialog tree contains message text, speaker metadata, code pointers, and directive metadata like multiple choice questions.

User progress through dialog trees is maintained in browser local storage. This includes the current node, directive answers, and whether nodes were visited. Separating mutable user state from immutable page content allows pages to be shared between learners without passing on user-specific data. It also allows \platforms to store user data locally, even for content that is deployed publicly.

EDBook documents use Markdown, a popular and simple markup language. EDBook uses an extended version of the Markdown syntax that enables custom generic directives, which we use for interactive assessments. For example, a user can author a multiple-choice question:

Whenever an author updates a message’s content, EDBook scans the message for any custom directives (including multiple-choice and code writing). It then creates a unique ID for each directive by concatenating the message’s ID with the directive type (e.g., multiple-choice) and the index of that directive. This way, users do not need to specify unique IDs manually (ericson2020runestone, ) and these IDs are resilient to wording changes and other small modifications. This unique directive ID is used as a lookup key to store and recall the user’s state for that directive (e.g., which multiple choice options are selected or the user’s current code). Although the EDBook platform does not have a formal plugin API, adding new directives only involves defining a React or JavaScript component that can (1) render HTML output, (2) serialize the component state, and (3) subsequently load that serialized state. Everything else (including tracking multiple directives’ states, determining when to render directives, and more) is handled by the EDBook platform. In the current implementation, directive states are entirely local and self-contained. This means, for example, that a multiple-choice question could not render differently depending on a user’s response to a different question.

Every message can have any number of larger associated code samples that get displayed when that cell is selected. These code samples are stored separately from the cell source. Authors can write code samples by simply selecting a cell and updating the code (e.g., adding files, removing files, or editing file content). Because these code samples can be lengthy and are likely to remain unchanged between multiple cells, the EDBook platform stores code changes between cells, rather than the full code. This helps reduce the file size of the resulting .dpage files. As Figure 8 illustrates, every message (cell) contains a list of textual diffs. These represent additions, removals, and unmodified lines relative to prior versions. Whenever a modification that may change the code is made (e.g., a cell is moved or deleted; or the code associated with a cell is changed), the code state for every cell is re-constructed and the code diffs are re-derived.

OpenAI’s ChatGPT Application Programming Interface (API) powers contextual LLM interactions. Readers can choose which model to query (GPT-3.5 by default). When users query the LLM, the current dialog context is combined with their question to generate a response. Constraining LLM prompts with dialog context mitigates unhelpful responses.

Uploaded images and other media are converted to binary data (UInt8Array), associated with a filename, and stored as part of the .dpage file (in the media field, as Figure 8 illustrates). Any deictic code references are stored as additional cell metadata, separate from the cell’s source.

As Figure 2 illustrates, \platforms exist in a continuum between less constrained tools (e.g., open-ended LLMs) and more constrained tools (e.g., pre-written linear dialogs). Although its design enables freeform user queries, the design nudges readers towards the pre-written narrative rather than self-directed exploration. \platforms were designed this way partially because of concerns about the accuracy and usefulness of information generated by LLMs—we prioritize presenting messages that we can guarantee to be accurate and pedagogically productive. As LLM capabilities (and our trust in them) continue to improve, future systems could instead consider nudging readers towards open-ended interactions and emphasizing LLM-generated content. Future work could also involve fine-tuning LLMs to improve their ability to generate accurate and contextually relevant outputs that align with specific learning goals. Further, the EDBook design could also be adapted to use LLMs to generate additional assessments and interactive content.

Further, although content authors can use LLMs to pre-generate content, we found that authoring branched narratives and coding challenges requires significant effort. Even if authors use LLMs, it can take significant effort to engineer prompts that produce high-quality content. Semi-automating content generation through AI could make authoring more scalable. However, safety mechanisms will also be needed to address potential issues around bias, misinformation, and harmful content from LLMs.

In its current design, readers navigate EDBook content individually. However, readers would likely benefit and learn from other readers’ inquiries and interactions with EDBook content. For example, future versions could give readers the option to share their open-ended interactions with other learners. These shared user-generated narratives could be curated to ensure that their content is appropriate, accurate, and high-quality. Reputation systems could surface high-quality user contributions and community voting or experts could identify helpful perspectives. User-generated content might even be able to ultimately replace the original dialog-tree to create community-written narratives that continue to evolve and improve over time.

Many aspects of the EDBook platform’s design could apply to domains beyond programming. For example, dialogs could guide readers through advanced UIs or creative and open-ended domains. To start, future work could replace EDBook’s code panel with applications that are appropriate for the given domain. With different prompting strategies, users could author customized tutorials for their own domain of work.