An Augmented Reality Platform for Introducing Reinforcement
Learning to K-12 Students with Robots

Artificial intelligence (AI) is playing a more prominent role in our daily life and has positively contributed to many fields and industries (Fosel et al. 2018; Zhou, Li, and Zare 2017). Reinforcement Learning, in particular, has generated great interest over the past decade but still suffers from the challenge of sample complexity. To alleviate that issue, recent works have focused on how humans can assist robots with direct feedback (e.g., demonstrations or advice) during the learning process (Ritschel et al. 2019; Yu and Short 2021). Yet, the robot’s inner workings and representations remain hidden from the users, which can result in a mismatch between the human’s mental model of the robot’s learning process and the actual inner workings on the robot. This paper proposes a tool based on Augmented Reality (AR) for the purpose of establishing common ground between a learning robot and its human teachers, with the goal of improving the efficacy of teaching and helping the human understand what the robot is learning.

Our primary application is in AI education at the K-12 level. Reinforcement learning (RL) has contributed to many fields (Kiumarsi et al. 2018; Fosel et al. 2018) but has received little attention in K-12 AI education(Coppens, Bargiacchi, and Nowe 2019). However, basic RL concepts such as rewards for doing the right thing are close to our perception of human learning (Sutton and Barto 2018) and therefore may be easy for K-12 students to understand. Robots have been successfully used in K-12 STEM education. (Petrovič 2021; Laut, Kapila, and Iskander 2013). Some researchers have applied educational robots on teaching AI to undergraduate design students, improving students’ engagement and lowering the barrier to entry in AI (van der Vlist et al. 2008). One limitation of using physical robots is the lack of an intuitive way for users to visualize the AI training process and the robot’s perception of the environment. We address this challenge by providing an AR tool that enables the user to visualize the robot’s learning process and to actively assist in the training of the robot.

Augmented reality (AR) is a technique to enhance the real world environment with virtual visual components, sounds or other sensory stimuli. By combining digital elements to a person’s perception of the real world, AR is powerful on providing immersive experience and visualizing abstract concepts in physical world. Compared with virtual reality, AR does not require substantial hardware and can be easily implemented on different platforms that are easy to access including mobile phone, tablet, laptop, etc. In recent years, AR has been applied in various fields and industries including education (da Silva 2019) and human robot interaction (HRI) (Hennerley, Dickinson, and Jiang 2017) which we would further introduce in the next section.

In this paper, we combined an AR interface with a LEGO® SPIKE Prime robot and designed a treasure hunting RL problem to introduce RL concepts to middle and high school students. The AR application can be easily installed on mobile devices (as showed in Figure 1) and the communication between the AR application and the robot is completed through an ESP8266 WiFi module connected to the robot. From the interface, the user can visualize internal RL representations and data structures. It also allows users to interpret key information related to training in an intuitive way. Since training a robot in physical world required longer time than training a virtual agent, we lowered the difficulty of our RL activity and gave users access to provide some instructions to the robot to decrease the sample complexity. The activity covers the following aspects of RL: 1) Key RL concepts, including state, action, reward; 2) Exploration and exploitation; 3) Q-table and how robots make decisions based on it; 4) Episodes and termination rules; and 5) Human-in-the-loop training. In the following sections we will unfold technical details and discuss potential applications of our system.

In the past ten years, AR has gradually become a popular tool in K-12 education. Researchers have applied AR technique on teaching robotics (Cheli et al. 2018), physics (Ibáñez et al. 2014), arts (Huang, Li, and Fong 2016), etc. And results from these studies showed that AR could not only increase students’ engagement, concentration and learning outcome, but also reduce the difficulty of learning. However, currently AR is rarely used in K-12 AI education. Some researchers developed a virtual reality platform that could introduce RL to K-12 students in a digital world (Coppens, Bargiacchi, and Nowe 2019).

AR also contributed to many HRI research including robotic teleoperation (Lee and Park 2018), robot debugging (Ikeda and Szafir 2021), etc. Related work has proposed an AR visual tool named “SENSAR” for human-robot collaboration (Cleaver et al. 2020), which was also deployed on a LEGO EV3 robot for educational purposes. The learning platform was an interface for students to visualize sensor values and control motors. In our research, we want to embed the learning experience into a game to make it more engaging, as other researchers also report greater engagement when the RL learning experience is centered around a game (Taylor 2011).

In Reinforcement Learning (RL), an agent has to learn how to act based on scalar reward signals detected over the course of interaction with its environment. The agent’s world is typically represented as a Markov Decision Process (MDP), a 5-tuple $<\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R,\gamma}>$, where $\mathcal{S}$ is a set of states, $\mathcal{A}$ is a set of actions, $\mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\Pi(\mathcal{S})$ is a transition function that maps the probability of moving to a new state given action and current state, $\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ gives the reward of taking an action in a given state, and $\gamma\in[0,1)$ is the discount factor. We consider episodic tasks in which the agent starts in an initial state $s_{0}$ and upon reaching a terminal state $s_{term}$, a new episode begins.

At each interaction step, the agent observes its state, and chooses an action according to its policy $\pi:\mathcal{S}\rightarrow\mathcal{A}$. The goal of the agent is to learn an optimal policy $\pi^{*}$ that maximizes the long-term expected sum of discounted rewards. One way to learn the optimal policy is to learn the optimal action-value function $Q^{*}(s,a)$, which gives the expected sum of discounted rewards for taking action $a$ in state $s$, and following policy $\pi^{*}$ after:

Q-learning (Watkins and Dayan 1992) is a common algorithm used to learn the optimal action-value function, where the Q-function is initialized arbitrarily (e.g., all zeros) and upon performing action $a$ in state $s$, observing reward $R$ and ending up in state $s^{\prime}$, the Q-function is updated by:

RL often requires large amounts of interaction with the environment. To speed up learning, researchers have proposed human-in-the-loop RL methods, e.g., learning-from-demonstration (LfD), where human teachers take over the action selection step, often providing several trajectories of complete solutions before the agent starts learning autonomously (Schaal 1997). In a related paradigm (Griffith et al. 2013), the agent can seek “advice” from its human partner, e.g., what action to select in a particular state for which the Q-values are thought to be unreliable due to lack of experience. One of the goals of our system is to visualize the RL agent’s knowledge and learning process as to provide information that makes human teachers more effective.

Our system seeks to illustrate RL in an intuitive way as to provide students with a more immersive learning experience. There are three main components in our system, a LEGO SPIKE Prime robot, an AR application on a mobile device, and an ESP8266 WiFi module that is used for communication. We also designed a treasure hunting activity based on our platform in which students could train the robot to find treasure and escape through the exit.

We proposed an AR robot platform for establishing common ground between a learning robot and a human user. We designed a treasure hunting activity based on it to introduce RL concepts to middle and high school students. The main significance of this project is to present an approach that can provide students a more interactive and immersive AI learning experience by combining physical robots with a virtual AR interface. We described the different components of our system and the RL activity we designed. We expect our platform can benefit other researches on K-12 AI education and provide some help on how to use AR in the visualization of human-in-the-loop AI training of robots.

We are designing a pilot study with students this autumn to evaluate their learning outcome and get feedback to improve our platform. We plan to separate students into two groups, one group use our platform to learn AR and another group learn the same concepts through an online platform we designed before (Zhang et al. 2021). The learning outcome of RL would be measured through a set of pre- and post- test. We would also compare the usability of two approaches and evaluate the value of AR and educational robots in K-12 RL learning. We are also working on constructing both easier and harder problems and create a storyline to let students explore them in a sequence. Finally, we also believe that visualizing the learning robot’s internal representation may help human teachers be better at providing feedback to the robot and we plan to evaluate this hypothesis with an additional study.

The research described in this paper was supported in part by a gift from the Verizon Foundation and LEGO Education.