HoloBoard: a Large-format Immersive Teaching Board based on pseudo HoloGraphics

Jiangtao Gong Lenovo ResearchBeijingChina [email protected] , Teng Han Institute of Software Chinese Academy of SciencesBeijingChina [email protected] , Siling Gong Lenovo ResearchBeijingChina [email protected] , Jiannan Li University of TorontoTorontoCanada [email protected] , Siyu Zha Tsinghua UniversityBeijingChina [email protected] , Liuxin Zhang Lenovo ResearchBeijingChina [email protected] , Feng Tian Institute of Software Chinese Academy of SciencesBeijingChina [email protected] , Qianying Wang Lenovo ResearchBeijingChina [email protected] and Yong Rui Lenovo ResearchBeijingChina [email protected]

(2021)

Abstract.

In this paper, we present HoloBoard, an interactive large-format pseudo-holographic display system for lecture based classes. With its unique properties of immersive visual display and transparent screen, we designed and implemented a rich set of novel interaction techniques like immersive presentation, role-play, and lecturing behind the scene that are potentially valuable for lecturing in class. We conducted a controlled experimental study to compare a HoloBoard class with a normal class through measuring students’ learning outcomes and three dimensions of engagement (i.e., behavioral, emotional, and cognitive engagement). We used pre-/post- knowledge tests and multimodal learning analytics to measure students’ learning outcomes and learning experiences. Results indicated that the lecture-based class utilizing HoloBoard lead to slightly better learning outcomes and a significantly higher level of student engagement. Given the results, we discussed the impact of HoloBoard as an immersive media in the classroom setting and suggest several design implications for deploying HoloBoard in immersive teaching practices.

teaching board, immersive learning, large-format display, hologram, mixed reality

^†^†copyright: acmcopyright^†^†journalyear: 2021^†^†copyright: acmlicensed^†^†conference: The 34th Annual ACM Symposium on User Interface Software and Technology; October 10–14, 2021; Virtual Event, USA^†^†booktitle: The 34th Annual ACM Symposium on User Interface Software and Technology (UIST ’21), October 10–14, 2021, Virtual Event, USA^†^†price: 15.00^†^†doi: 10.1145/3472749.3474761^†^†isbn: 978-1-4503-8635-7/21/10^†^†ccs: Human-centered computing Mixed / augmented reality

Refer to caption — Figure 1. HoloBoard in a regular lecture-based class (recorded by an 8K Panoramic Camera)

1. Introduction

Recent advances in understanding new instructional technologies (e.g., interactive whiteboard, VR, AR) indicate a need to rethink how to incorporate active learning into traditional lecturing (Balta and Duran, 2015; Karsenti, 2016; Ibrahim et al., 2018; Vazquez et al., 2017; Holstein et al., 2018; Aslan et al., 2019). With rapid growing and diverse applications, emerging displaying and interaction technologies have seen potential to scale in classroom adoptions. Indeed, their role in education is attracting increasing attention and is continuously verified in various contexts, e.g., virtual experiment simulation (Apostolellis and Bowman, 2014; Ferracani et al., 2014; Freina and Ott, 2015), AR-based word learning (Ibrahim et al., 2018; Vazquez et al., 2017). However, traditional immersive hardware such as Head Mounted Displays (HMDs) are not entirely suitable for the promoting and applying AR/VR in lecture-based classroom environments. These hardware might cause visual and body discomfort while wearing them, difficulties in facilitating collaborations, dangers of collision during multiplayer mode, and the costly operation/maintenance of the headsets (Radu, 2014).

This paper paid attention to teaching boards, which have been used as essential venues to display curriculum materials and help teachers communicate with a large audience in classrooms. Teaching boards have been evolving towards being more functional and interactive in enhancing pedagogical performance (Muttappallymyalil et al., 2016). In this work, We present a novel concept of HoloBoard, a pseudo holographic based immersive teaching board system with a large-format transparent display. Pseudo holographics are of interest in science, retail and exhibition applications in recent years. It offers a pseudo 3D image to see near its surface by the reflected light of the dedicated projector. HoloBoard exploits such capabilities and aims to make lecture-based classes more attractive by blurring the boundary between physical classroom and digital learning content, enabling immersive content display. In comparison with AR/VR, HoloBoard is advantageous in that its setup does not require everyone to wear an HMD, and offers rich interaction possibilities such as co-located mixed reality collaboration with face-to-face communication and natural eye-contact. Our work to reach the goal is two-fold: 1) interaction designing of HoloBoard, and 2) verifying its effect in engaging students in lectures.

The unique visual experiences of watching pseudo 3D images empower us to design a set of interaction techniques that are potentially valuable for lecturing in classroom. To do so, we adopted a user-centric design thinking process (e.g., interview, brainstorming) to excavate teachers’ requirements for an immersive teaching board. With the resulting design considerations, we developed the prototype of HoloBoard associated with advanced interaction techniques ranging widely from immersive presentation to augmented role-play to see-through virtual interactions. Next, to understand how the presentation and interaction of HoloBoard could help teaching and engaging students in lectures, and to give insights on practical deployment of HoloBoard, we carried out a second study in two actual classes with HoloBoard and a regular digital whiteboard, respectively. We used pre-/post- knowledge tests and multimodal learning analytics techniques to measure students’ learning outcomes and learning experiences. Results showed that HoloBoard classes created a significantly more positive learning atmosphere, lead to significantly higher student engagement, and resulted in slightly better learning outcomes. Given these, we discussed the impact, design implications and practices of such immersive media in classrooms.

The paper made the following contributions: i) HoloBoard as a novel large-format immersive teaching board in classroom, built upon pseudo holographic system; ii) design of interaction techniques with HoloBoard that are potentially valuable for lecturing and engaging students; iii) a formal experiment that compared HoloBoard with a regular digital whiteboard and measured students’ learning outcomes and learning experiences; iv) insights on deploying HoloBoard in classrooms.

2. Related Work

Our related work include works which inspired the HoloBoard concept such as immersive teaching technologies, interactive technology for learning, in-class engagement, and multimodal learning analytics.

2.1. Immersive Teaching Technologies

Lecture-based class leverages the naturalness of spoken communication and creates a social situation that makes both the lecturer and the audience engaged (Charlton, 2006). The role of teaching boards is critical in lecturing activities. The importance of teaching boards in the education domain has been thoroughly discussed, especially in displaying curriculum materials and enhancing pedagogical performance (Gran, 2018). Over the past decades, a significant body of research focused on the impact of Interactive Whiteboards (IWBs) (Betcher and Lee, 2009) on learning (Morgan, 2008; Yang et al., 2012; Balta and Duran, 2015; Karsenti, 2016), noting their positive effects on student engagement, behavior (Morgan, 2008), and learning outcomes (Yang et al., 2012).

Nowadays, immersive displaying technologies are getting prominent in educational applications (Apostolellis and Bowman, 2014; Ferracani et al., 2014; Freina and Ott, 2015; Ibrahim et al., 2018; Vazquez et al., 2017). Virtual reality (VR) technology is entering mainstream consumer markets through products such as HTC Vive and Occulus Rift/Quest. Both educators and learners share positive attitudes towards using VR for educational objectives (Mikropoulos and Natsis, 2011) and numerous studies have reported the benefits of VR in education, which include improving time-on-task (Huang et al., 2010), enjoyment (Apostolellis and Bowman, 2014; Ferracani et al., 2014), motivation (Freina and Ott, 2015; Sharma et al., 2013), and long-term retention (Rizzo et al., 2006). Researchers have found that immersive VR has an advantage over the desktop systems when the tasks involved “complex, 3D, and dynamic” content (Mikropoulos and Natsis, 2011).

Augmented reality (AR) technology has made significant advances over recent years. Many previous AR systems for education focused on using HMDs, mobile phones, or projectors to enhance learning equipment or environments for students in studying a wide range of subjects, such as language (Ibrahim et al., 2018; Vazquez et al., 2017), chemistry (Fjeld et al., 2007; Song et al., 2011), and mechanic engineering (Ibáñez et al., 2014; Enyedy et al., 2012). Some work explored augmenting teachers’ views to assist them with classroom routines (e.g. evaluating student’s performance (Holstein et al., 2018)). Overall, AR can support pedagogical processes (e.g., providing scaffolding to students) and promote students’ engagement (Aslan et al., 2019).

HoloBoard is built upon an alternative displaying technology, namely pseudo holographics, to enrich the interaction vocabularies around digital teaching boards for more substantial engagement.

2.2. Interactive Technology for Learning

The human-computer interaction and learning science community have long recognized the potential of interactive technology for increasing engagement, as they vastly broaden the possibilities for students to explore class materials and actively participate in learning processes. Simple student response systems (also know as ‘clickers’), which offer a small number of buttons for students to provide answers, were found to help students better get involved in large enrollment courses (Trees and Jackson, 2007). General purpose computing devices such as tablets support students participating in richer kinds of learning activities, including drawing (Couse and Chen, 2010), educational games (Furió et al., 2013), and simulated labs (Nedungadi et al., 2013). Experiments have confirmed their effects in improving engagement (Couse and Chen, 2010) and learning outcomes (Haßler et al., 2016). In addition, experimental interfaces such as tangible user interfaces (Xie et al., 2008; Schneider et al., 2016) and robots of varying form factors (Özgür et al., 2017; Tanaka and Matsuzoe, 2012) have been applied in classroom settings and shown their benefits for increasing student engagement, enjoyment (Özgür et al., 2017; Xie et al., 2008), and sometimes learning outcomes (Tanaka and Matsuzoe, 2012; Schneider et al., 2016).

A large body of research focused on encouraging students’ participation through whole-body movements (Kang et al., 2016; Lindgren and Johnson-Glenberg, 2013; Gallagher and Lindgren, 2015; Kontra et al., 2015; Gelsomini et al., 2020). These efforts were supported by an embodied perspective of cognition, which asserts that human cognition is affected by the body’s interaction with the physical world (Pecher and Zwaan, 2005; Kontra et al., 2015). Additionally, whole-body interaction promotes student engagement as it offers immersive experiences (Adachi et al., 2013), facilitates social interaction in classrooms (Kang et al., 2016), and possibly transforms learning activities into performances (Gelsomini et al., 2020). For example, Kang et al. explored combining immersive displays, whole-body interaction, and physiological sensing for teaching children knowledge about human body (Kang et al., 2016). They found that the system enabled overall high levels of engagement and enjoyment, lively interaction between students, and strong learning potential.

Leveraging the unique affordances of transparent holographic displays, HoloBoard enabled a new array of classroom activities to engage students, such as instructor role playing, large-format virtual demonstration and multi-user game, and double-sided interactions between students/instructors or students/students.

2.3. In-class Engagement and Multimodal Learning Analytics

Many aforementioned studies (Adachi et al., 2013; Couse and Chen, 2010) have focused on using educational technology to improve student engagement because high levels of engagement can compensate for low academic achievement, negative affective states (e.g., boredom), and high dropout rates among students (Fredricks et al., 2004; Gao et al., 2020).

It is widely accepted that engagement is a multifaceted concept with three dimensions: behavioral (active participation and involvement in activities), emotional (positive reactions and feelings towards teachers and work), and cognitive (efforts and concentration on completing work) engagement (Fredricks et al., 2004; Gao et al., 2020; Stroud, 2015). There are various ways to measure engagement in educational related studies. For example, the widely agreed upon student engagement signs for behavior engagement include upright or close posture, gaze and focus on the teacher or the task, and active participation in classroom discussion and they have been captured in the coding schemes used in (Beşevli et al., 2019; Hsieh et al., 2015). For emotional engagement, despite self-report, students’ affective states are commonly used for analysis, which can be based on arousal and valence feature analysis (Di Lascio et al., 2018; Alyüz et al., 2016). Classroom discourse analysis, cognitive level of speech, and the interactive nature of conversations have been used for measuring cognitive engagement (Stroud, 2015; Smart and Marshall, 2013; Zhu, 2006). More specifically, previous studies in the field of education and human-agent interaction have investigated various discourse attributes, including total speaking turns taken, total words spoken, and mean words per sentences (Șerban et al., 2017; Stroud, 2015; Black et al., 2009).

Considering the rich signs of student engagement, we adopted multi-level multimodal learning analytics to analyze the classroom data. Multimodal learning analytics in education is an emerging branch of learning analytics, and it typically analyzes natural synchronized communication modalities, including speech, gestures, facial expressions, gaze, and embodied actions (Oviatt, 2018; Bateman et al., 2017; Worsley and Blikstein, 2015). Analysis of multimodal data can take place at multiple levels (Oviatt, 2018). Taking speech as an example, it can be analyzed at the signal level (e.g., frequency and loudness), activity level (e.g., number of words spoken), or other levels (Oviatt, 2018).

In addition, Oviatt et al. (Oviatt, 2018) suggested that such multi-level multimodal learning analytics can support a more comprehensive understanding of the complex learning process, such as the impact associated with implementing a new educational technology. Therefore, we hope to obtain a deep view of the impact of HoloBoard on students’ in-class engagement by utilizing multi-level multimodal learning analytics.

3. HoloBoard

HoloBoard aims to provide unique and immersive lecturing techniques with large-format pseudo holographics to engage students in learning processes. It was developed based on pseudo holographic projection, which was scalable in size and of a formfactor close to typical teaching board. We conducted interviews with 6 professional teachers to understand their teaching activities and requirements, which were then mapped to ten interaction techniques of HoloBoard. We built a working prototype of HoloBoard with the resulting design considerations.

3.1. Concept Design

To establish an initial understanding of traditional teaching activities and to better understand how the concept of HoloBoard can be incorporated into classrooms, we conducted a series of semi-structured interviews with 6 school teachers (4 females, average of 5.58 years of teaching experience in English, Math, Physics or Music in primary school, middle school or high school). The interviews with the teachers were aimed to understand their teaching process and activities in classroom. Besides, in the interviews, we asked questions regarding their teaching strategies, multimedia equipments and supplementary materials used in class, their preferred teaching methods to handle different types of content, and approaches to interact with students in teaching.

All the interviews were audio-recorded and transcribed. In total, 350 minutes of audio materials were collected. The interview transcripts were analyzed using the open coding method (Corbin and Strauss, 2014). Afterward, all codes were transcribed on sticky notes and arranged based on the teaching process via affinity diagramming (see Fig. 2).

From the affinity diagram, we obtained the main flow of a typical lesson described as following steps: pre-class preparation (designing the course content and preparing physical and digital materials), warm-up (sharing interesting audio or video clips), lecturing (slides sharing, writing, performing, and physical demonstration), teacher-student interaction (answer questions and students’ presentation), and review (write down key takeaways on the blackboard or slide decks). During lecturing, we found that all the teachers have two basic requirements of teaching board: i) to display electronic teaching content of various forms, such as slides, text and video display; ii) to support writing and freehand drawing.

In addition to basic requirements, we found the teachers have advanced requirements to guide students’ imagination and enhance their attention. For instance, the teachers who teach primary school students tend to incorporate body language into teaching: T5 reported that when she taught students arithmetic: “sometimes I also design exaggerated performances or teach them some specific actions to remember this formula.” T6 used movement to deepen students’ understanding of rhythm: “I will tell them what kind of scenes they can imagine when they hear this music and then lead them to walk in the classroom following the rhythm of the music.” Besides, teachers for classes that require demonstrations used tools and proxies such as physics lab equipment (Physics teacher), toy blocks (Math teacher), and musical instruments (Music teacher). These physical demonstration tools also could encourage students to engage, explore, experiment, and learn actively.

3.2. Interaction Techniques with HoloBoard

We incorporated the resulting findings from the interviews and the unique presentation properties of pseudo holographics into 10 interaction techniques that are potentially valuable for lecturing scenarios.

3.2.1. Immersive Presentation

The pseudo holographic projection has the ability to create the illusion of 3D objects in volumetric space. The projection space is large, bringing immersive visual experiences to the audience. These features give HoloBoard a bunch of novel capabilities for large-format immersive content display (Fig. 3a), uniquely suitable for drawing large audiences’ attention with immersive visual presentations. This vivid and engaging visual experience is unique to the HoloBoard and can not be achieved with regular digital board.

3.2.2. Lecturing Behind the Scene

HoloBoard allows teachers to lecture behind the teaching board, with digital sketches and writings floating between them and the students. Previous researches (Ishii and Kobayashi, 1992; Saquib et al., 2019) have demonstrated that real-time graphic overlays do not interfere with the natural conversational style of the presenter, allowing users to effortlessly enhance their communication with the audience, the HoloBoard takes advantage of this flexible and highly adaptable communication style. One technical challenge posed by the transparent interface are the inverted perspectives of the presenter and the audience. We designed a method of text reversal in order to display the correct text for the presenter and the audience simultaneously (Fig. 3b).

3.2.3. Role-play

With the transparent display, students can see pseudo 3D images without losing sight of the presenter behind the HoloBoard. Thus, HoloBoard supports live action role play, during which teachers or students literally act out fictional roles within a narrative (Fig. 3c). Users behind the board are visually augmented by a digital the avatar overlay, which is animated in real-time according to the users’ movements. Users directly drive the generation or motion of graphics by performing specific poses or gestures. This form of interactive and collaborative storytelling gives an immersive and complex sense of a narrative engagement to students.

3.2.4. Afterimage

HoloBoard integrated techniques of motion visualization in order to convey complicated actions. In specific, the system captures the motion of the presenter, and produces a visualization of the spatiotemporal representations of the presenter as she moves through the space or gestures (as shown in Fig 3d). For instance, when teaching martial arts, taekwondo, dance, etc., the teacher can freeze his current action at any time as an afterimage. In this way, the teacher can give a detailed explanation of these movements enabling students to perform imitation exercises step by step, as necessary.

3.2.5. Augmented Dashboard

Research indicates that teachers are interested in having access to real-time status of students (Holstein et al., 2018; Aslan et al., 2019). HoloBoard supports such applications by displaying students’ learning analytic information (e.g., test score, emotion) to teachers, undated in real time. With HoloBoard, the student’s name and other relevant information is associated with each student, enabling teachers to modify lesson plans spontaneously in accord with the changing needs of students, as indicated by various features of the augmented overlay (e.g., changing name tag’s color). Additionally, the teacher can give interactive markers (e.g., praise) to individual students by pointing to and clicking on the student’s name tag, as shown in Fig. 3e.

3.2.6. Co-located Mixed Reality Collaboration

Due to its large-format and transparency, HoloBoard supports high quality co-located multi user interactions. HoloBoard also supports co-located users with face-to-face communication. Typically, with a digital whiteboard, two users both have to face the screen, obscuring face-to-face communications, e.g., eye-contact is impossible. HoloBoard overcomes these obstacles of face-to-face communication, allowing users to stand on each side of the board, respectively, and shared a common use space with fully interactive contents. Because users can still see each other clearly, their communication remains natural and spontaneous, enhancing the co-located mixed reality collaboration, as shown in Fig. 3f.

3.2.7. Physical to Virtual Interaction

In typical classroom, students barely have the chance to interact with the digital contents on screen when they are seated. HoloBoard provides opportunities to interact with the screen in multiple ways. For instance, students in front of the screen can throw a soft ball onto the screen to make a selection or play games. Such method can be further extended by rendering a virtual ball that moves along the trajectory of the real one, thus letting the physical actions reach into the virtual world. Previous researches(Jones et al., 2013; Jones et al., 2014) also used surrounding projectors to make physical object interact with virtual content and create an immersive gaming experience in a room space. This helps expanding the interaction space of the students and creating realistic experiences with simulated scenes as shown in Fig. 3g.

3.2.8. 3D Modelling with Elastic Haptic Feedback

The projection screen of HoloBoard is made of gauze that has a certain elastic property. Previous work (Watanabe et al., 2008; Han et al., 2014; Müller et al., 2014) have shown that operations in Z-direction on the screen and the haptic feedback of touching on elastic surfaces helped improving 3D content presentation and maneuvering experience, for instance, improving depth perception. HoloBoard is capable of supporting teachers and students to perform 3D modeling tasks on the screen, while gaining the haptic experiences like hands reaching into the scene as shown in Fig. 3h.

3.2.9. Holographic Telepresence

Holographic telepresence is another scenario that fits the use of HoloBoard (Paredes and Vázquez, 2019; Aman et al., 2016; Luévano et al., 2015). This helps extending the face-to-face experience of in-class instruction to remote areas. In this case, it can serve as a medium for remote education and mixed remote panels, displaying remote teachers vividly in the classroom as shown in Fig. 3i.

3.2.10. Mirror World

HoloBoard can serve as a medium for a co-located virtual mirror world, displaying digital twin of the physical world. HoloBoard can elaborate on the learning benefits of the whole-body interaction based on mirror world, which has been shown effective for learning knowledge about the human body (Kang et al., 2016), physical experience in science learning (Kontra et al., 2015), and other embodied learning. Different from the telepresence where users are remotely located and can not observe the environment and people on site, the mirror world technique allows the teachers and students to see themselves interacting with digital objects and be aware of the surroundings ( Fig. 3j).

3.3. Prototype Development

We built a working prototype of HoloBoard, with which we developed the demo applications enlightened from the aforementioned interaction design.

3.3.1. Hardware and Software

The system of HoloBoard was developed in a classroom, and integrated the displaying and tracking components (Fig. 4). In terms of display, we used a 4m×3m large-format projected screen made of translucent (semi-transparent and reflective) material to display digital curriculum material in the front of the classroom. Behind the projected screen is the teacher’s stage. Moreover, to further immerse students into the scene, a light-absorbing curtain is hung behind the stage, and 4 adjustable spotlights are put above the staging area to cast more light on the teacher. Thus, the observer can spot the people behind the screen while the projected screen displays.

In terms of motion tracking, we used 4 SteamVR 2.0 base stations at the 4 corners of the classroom, one on each corner. Two of them were in front of the projection screen while the other two behind the screen. We asked the users to hold the HTC Handler to interact with the HoloGraphics with hands, and wear 4 VIVE trackers on the head, waist, left and right feet, respectively, to track their position and body motions. The system including the communication servers, graphic rendering, and demo applications were implemented with Unity. Many interaction techniques designed for HoloBoard shared the same set of displaying, tracking setup.

3.3.2. Double-sided Rendering for HoloBoard

The key to creating an immersive holographics lies in real-time rendering of the position, size and angle of the content with reference to changes in the observer’s position and perspective. We achieved this by converting the spatial position into the camera coordinate system of Unity, and calibrating the orientation and scale of the digital contents to be rendered based on the user’s position. Experimentally a point in the center of the classroom was selected as the ”best” viewing spot for the students and teacher. It simplified the calculations while guarantying the rendering respond to the user’s position and movement accordingly.

3.3.3. Implemented Scenario

According to the interaction technique design in the former section, we implemented 12 scenarios for the working prototype as showing in Fig. 5 and our demo video.

(a)

Immersive Presentation by Displaying large 3D Content: When the teacher clicks the next/previous page on the controller, it will automatically jump to the next/previous slide.
(b)

Lecturing Behind the Scene: The teacher can manipulate the digital content flowing between the teacher and students by one or both controllers, with eye contact with students at the same time.
(c)

Writing Behind the Scene: The teacher can write freely with one controller behind the scene. The system displays the original text using a inconspicuous color for the presenter and generates another text reversal to display the correct text using a conspicuous color for the audience simultaneously.
(d)

Role-playing an Astronaut: The spatial positions of the controllers and trackers worn by the teacher are mapped to a 2D/3D avatar model with an animated skeleton structure. When the teacher moves, the 2D/3D avatar model will be rendered on the projected screen right in front of the teacher and copy the teacher’s movement. This function is accomplished by converting spatial coordinates into project coordinates based on students’ point of view. From the students’ view, the teacher is augmented by the avatar outfit, which can also be used to trigger more animations.
(e)

Step-by-step Teaching with Afterimage: The teacher can make a pose and click the button on the controller to freeze his/her image/avatar on the current page of HoloBoard.
(f)

Augmented Dashboard of Students’ Status; By tracking the position of the teacher’s eye and students, each student’s name tag and other real-time learning information will appear above each student’s head. Furthermore, the teacher can give marks (like praise) to a certain student by clicking on the student’s name tag.
(g)

Co-located Mixed Reality Collaboration: The user can pass a virtual object towards another user on the other side by the controller with naturally eye contact at the same time, such as playing tennis, collaborative manipulation;
(h)

Physical to Virtual Interaction: The students can throw a physical ball into a virtual world displayed by HoloBoard. We tied a vive tracker to the ball for tracking. When the physical ball contact the screen, the system create
(i)

3D Modelling with Elastic Haptic Feedback: The user can create a new model in Z-direction on the screen by hold the controller into the HoloBoard.
(j)

Holographic Telepresence: The remote user can be photographed by the camera in real-time in front of the green screen, and after removing the background, it will be displayed vividly on the HoloBoard.
(k)

Mixed Reality Panel: The local users can sit behind the HoloBoard and conduct a mixed reality panel, discussion with remote users’ vivid telepresence on the HoloBoard.
(l)

Mirror World: The users can be photographed/captured by a camera, and blended, re-positioned, and resized into the digital mirror scene displayed by HoloBoard.

4. Experimental Study

To investigate whether HoloBoard can actually facilitate children’s learning (e.g., engagement and learning outcome) and explore the additional value it can offer to a lecture-based class, a controlled experimental study was conducted.

4.1. Course Design

HoloBoard is designed as an essential teaching board, while having a form different from the normal interactive whiteboards. We conducted a participatory course design process inspired by (Penuel et al., 2007) to amplify the advantages of HoloBoard and design a course for experimental study.

Five professional teachers (all females, avg. teaching experiences = 4.2 years in Literature, STEM or Arts in primary school, middle school or high school) got invited. HoloBoard’s designers intensively conducted in-person and online participatory design workshops (over four formal workshops, each lasting 1-2 hours) within 6 months (Jul 2020 - Dec 2020). Moreover, we collected the syllabuses, courseware, video recordings of lectures, and student work from previous similar courses from these participating teachers to better understand user scenarios.

After rounds of discussion, teachers and researchers reached an agreement that topics about the outer space can be highly interesting to young students. According to Piaget’s theory of cognitive development (Huitt and Hummel, 2003), children aged 6-9 are at the Concrete Operational Stage. In this stage, in general, children’s abstract thinking skills are still developing and they can only apply abstract concepts, (e.g., time, space) in concrete real-life situations. Therefore, with the large-format and highly interactivity, HoloBoard has the potential for facilitating children’s abstract learning by displaying the course content (i.e, space) in an intuitive way, benefiting from 3D visual effects of planetary, interactive experiment platform, and situated role-play, etc.

Eventually, a STEM course (i.e. Space Exploration) targeting six- to nine-year-old students was designed. The course focused on the characteristics of the eight planets in the solar system. In the course, students were invited to explore the eight planets in the solar system in order to find a habitable planet. Students were guided to think about questions including ”why should humans explore Mars?” and “what kinds of planets are habitable?” The course adopted the inquiry-based science teaching method and was designed according to the 5E Instructional Model process (Duran and Duran, 2004).

Three main features of HoloBoard were adopted into the course: 1) the teacher can role-play as an astronaut to roam the solar system, and guide the students to learn the characteristics of the eight planets situationally; 2) the teacher can double-sided demonstrate the order of the eight planets interactively using virtual experiment simulation; 3) the students can double-sided play multi-user game to test their knowledge about the characteristics of the eight planets. In the normal group, to maintain the fairness of the comparison, the script of the classes was kept the same and the following adjustments to activities were made. For Feature 1, (AR-based role-pay), in the normal group, the teacher also verbally pretended to be an astronaut (asking the students to imagine that she was an astronaut) and thus kept the teaching script the same. For Feature 2 (double-sided demonstration), during learning the order of the planets, a similar demonstration of planet order was displayed on the interactive whiteboard in the control class (as depicted in Figure 7(b)(e)). For Feature 3 (double-sided multi-use game), while summarizing planetary habitability, the teacher mainly posed guiding questions (e.g., Is Mars a good place to live in?) and asked students to analyze from different dimensions (e.g., gravity and dangerous factors) in the two classes. The only difference is that in the normal group, a table summarizing the planet habitability was used while the interface of the VR multi-user serious game was used in the HoloBoard group.

4.2. Site Preparation

Besides HoloBoard setup, a traditional classroom with a regular interactive whiteboard (4K, 86-inch) was set up. Desks and chairs were arranged in five groups. To facilitate data gathering, six cameras were set up around the classroom. One 8K panoramic camera (TECHE360anywhere) was set up in the middle of the classroom. An audio recording device (iFLYTEK SR501) was also set in the experimental classroom. Furthermore, this research received IRB approval from our local institution.

4.3. Participants

36 participants (Female = 18, 15 in Grade 1, 15 in Grade 2, and 6 in Grade 3, avg. age = 7.06) were selected upon a pre-experiment survey and a pre-test knowledge test. Most participants reported being exposed to digital devices like projectors, mobile phones, tablets, and PCs before. Approximately 78% of the participants mentioned preferences for these smart digital devices. Moreover, the participants were fairly interested (4.3 out of 5) in space related topics. Their motivation, engagement, and learning self-efficacy on average were at a slightly high level (15.5 out of 20).

Besides, we invited one of the teacher subjects in the course design session to be our experiment teacher (T7, STEM teacher, 26 years old, female, master’s degree in Education, 2 years of teaching experience). This teacher teaching both HoloBoard group and normal group with same class script.

4.4. Procedure

We pairwise assigned the participant students (based on their gender, age, grade, performance on the pre-test, interests in related topics, learning attitude, and past experience with technology) to either a control group (normal group) subjected to the traditional practice based on the normal lecture-based class, or an experimental group (HoloBoard group) subjected to the practice within HoloBoard environment. In addition, we discussed the course outline with the teachers to further control their lecturing script for the two classes.

Before the experiment, parents/legal guardians of the participants all signed the informed consent release form. During the experiment, the two groups of students were taught the same 70-minute lecture featuring either the HoloBoard or the normal interactive whiteboard. The participants were seated in assigned small groups. After the experimental class ended, the experimenters conducted a one-on-one post knowledge test with each student participant. Each post-test took approximately 20 minutes.

A pre-/post- knowledge test (See Appendix) was designed, identical for all participants, to check students’ knowledge of class content before and after the lecture-based class. The test consisted of 12 questions of different complexity and types (four single-choice questions, two multiple-choice questions, and six open-ended questions), each of which covered one or more cognitive learning objectives of the class. The test was developed following the examples of usual primary school tests and was validated by the teachers. Each single-choice question is worth one point and each multiple-choice question is worth two points. The score for open-ended questions can range from 0 to 5 based on the rating rubric used in (Lui, 2018). Thus, the full score of the pre-/post knowledge test was 38.

4.5. Data Collection and Analysis

Our data came from three primary sources: 1) the pre- and post-knowledge tests, 2) audio recording and transcripts of the HoloBoard class and the normal class, and 3) video recordings of the two classes. The knowledge test was used for learning outcome comparison analysis and the other two were for student in-class engagement comparison analysis.

4.5.1. Pre-/post- Knowledge Test Data.

Students’ answers to the single/multiple choices questions in the pre- and post- knowledge tests were scored by two researchers based on the right answers provided by the teacher. In addition, students’ answers to the open-ended question were audio recorded, transcribed, and rated by two trained coders based on a rating rubric adapted from the Knowledge Integration rubric used in (Lui, 2018). Two trained coders independently coded all students’ responses. The inter-rater Kappa was greater than 0.70 (p ¡ .001). A third coder reviewed their ratings, and consensus were eventually achieved. Then the total scores of each each students’ pre- and post-tests were calculated and compared.

4.5.2. In-class Engagement Learning Data and Analytics.

The learning data included video and audio recordings and transcripts of the two classes.

Video Data and Analysis. The video recordings of the two classes were mainly used for student behavioral and engagement analysis by manual coding. The coding scheme was adapted and integrated from (Beşevli et al., 2019; Homer et al., 2018; Hsieh et al., 2015; Pas et al., 2015; Wagner et al., 2005), which pictured students’ reactive behavior to the two teaching devices and reflected students’ engagement in the classes. The three main categories of our codes are affective states (emotional engagement), classroom behavior and posture (behavior engagement). The detailed types and descriptions, and operational definition of students’ nonverbal behaviors are presented in Table LABEL:tab:NonverbalCode.

We selected three video clips from the HoloBoard and the normal classes, respectively, discussing the same topics (exploring planets’ characteristics, discussing the order of planets, and summarizing planetary habitability). While discussing the first two topics, in the HoloBoard group, the two key features of the HoloBoard—AR-based role-play and VR simulator demonstration—were used respectively. As noted in the course design section, these activities remained comparable because they used similar scripts and both involve elements of role play and made use of the same images. To keep the comparability between the two classes, the video clip of children playing the multi-user serious game on HoloBoard was excluded from the analysis. Furthermore, three random 10-second video clips were sampled and analyzed from the two classrooms’ six video recordings (see fig.7). The random sampling of short video clips has been widely used in prior literature for analyzing student interaction (e.g., (Groccia, 2018) and (Karsenti, 2016)). Each participant was analyzed through 90-second video data in which their nonverbal behavior was coded. All video clips were analyzed using the Noldus Observer XT software. While coding the video clips, the type of nonverbal behavior of each student mentioned in Table LABEL:tab:NonverbalCode was identified and marked by the coder in the Observer software, and the duration of each behavior was automatically calculated and computed by the software. To calculate the length of each behavior, our unit of time was defined as 0.1 seconds. Two trained coders coded each student’s behavior based on the coding scheme independently. Inter-rater reliability was conducted on 22% of the video data, where the inter-rater Kappa was greater than 0.89 (p ¡ .001).

Table 1. Manual Coding Scheme of Video Recordings

Categories	Types	Description
Affective States	High Arousal Positive Valence	Joy: happy, excited, aroused, concentrated, etc.
	Low Arousal Positive Valence	Calm: relaxed, satisfied, and content, etc.
	High Arousal Negative Valence	Anger: tense, annoyed, frustrated, etc.
	Low Arousal Negative Valence	Boredom: bored, depressed, miserable, etc.
Posture	Close Posture	Moving closer to or leaning toward the whiteboard/HoloBoard
	Neutral Posture	Sitting straight up
	Leave Posture	Moving body away or leaning away from the whiteboard/HoloBoard
Behavior	Positive Behavior	Engaged behavior: answering questions, raising hands, looking at the pupil speaking, applauding, etc.
	Normal Behavior	Normal behavior: listening, looking at the screen/teacher, writing, etc.
	Misbehavior	Disengaged behavior: chatting, looking away etc.

Audio Data and Analysis. The HoloBoard and the normal groups’ classroom discourses were recorded, fully transcribed, and analyzed. The audio recordings were used to conduct an acoustic analysis of frequency and loudness as references to student emotional engagement. The acoustic analysis was performed in pyAudioAnalysis. To better portray the change of student engagement and compare the results of the two groups, we selected and computed the first quarter of data of the two classes as a baseline.

Transcript Analysis. The audio recordings of the two classes were fully transcribed. The transcripts were firstly used to compare the teachers’ speech type in order to confirm the effectiveness of controlling teachers’ lecturing script. The coding scheme of teachers’ speech was mainly adapted from FIAS (Flanders, 1970). As shown in Table LABEL:tab:VerbalCode, teachers’ initiated language was categorized into three main types: lecturing (e.g., giving information), directing (giving direction or instructions), and asking questions. Then, questions were further divided into close- or open-ended questions as previous research has shown that teachers’ questions have influence on student cognitive engagement (Smart and Marshall, 2013; Murcia and Sheffield, 2010; Lee and Kinzie, 2012).

Focusing on students, we used the transcripts to analyze cognitive and emotional engagement by both manual coding and algorithm processing. For cognitive engagement analysis, firstly, students’ speech types (i.e., statement and questions) and cognitive levels were coded. For student speech type, the coding scheme was adapted from the linguistic expression category of the scheme in (Mulder et al., 2002). For the cognitive level analysis, Bloom’s taxonomy of cognitive domain (Bloom et al., 1956) was used and thus the cognitive level of students’ responses were categorized into: remembering, understanding, applying, analyzing, evaluation, and creation. Four trained coders coded the transcript of the two classes, and the inter-coder Kappa was greater than 0.81 (p ¡ .001).

Moreover, the whole transcripts were used to compare the attributes of discourses as references to cognitive engagement. The discourse attributes we investigated include: (a) the number of speaking turns, (b) the number of times each student spoke, (c) the number of words and sentences each student spoke, and (d) the length of students’ sentences and rounds of dialogue, which were adapted from the related studies (e.g., (Stroud, 2015; Șerban et al., 2017; Black et al., 2009)).

To measure emotional engagement, we captured students’ exclamations (e.g., “Wow” and “Oh My God”), which was captured to assess students’ engagement in (Beşevli et al., 2019). In addition, we also used Baidu AI Cloud (Tian et al., 2020) to perform semantic analysis as indicators of emotional engagement.

Table 2. Manual Coding Scheme of Transcripts

Speaker	Categories	Types	Description
Teacher	Speech Type	Lecturing	Giving information or opinion
		Directing	Giving directions
		Asking Close-ended Question	Calling for a single response
		Asking Open-ended Question	Calling for board responses
Student	Speech Type	Assertion	Statements of facts, opinion, choices, etc.
		Question	Request for information
		Exclamation	Sound/Words expressing strong emotion (e.g., “Wow” and “Pff”)
	Cognitive Level	Remembering	Reciting and memorizing labels
		Understanding	Relating and organizing previous knowledge
		Applying	Applying information into a new situation
		Analyzing	Drawing connections among ideas
		Evaluation	Examining information and make judgement
		Creation	Creating new ideas using what has just been learned

4.6. Results

We firstly compared the teacher’s script in the HoloBoard and normal classes. Teachers’ questioning accounted for approximately six percent of the speech, among which 70% of the questions were close-ended questions in both classes. The results suggested there were no major differences in the types of speech initiated by the teacher between the two classes. Then we describe the results of students’ learning outcomes and engagement as the following.

4.6.1. Learning Outcomes.

We conducted Mann-Whitney U test to compare the knowledge pre-test results as baseline between the normal and HoloBoard groups. The result suggested there was no significant differences between the two groups ( p = 0.46).

Next we compared learning gains and performances between the two groups. The analysis results of learning outcomes are presented in Table LABEL:tab:testscore. For the learning gains, the post-test score of the HoloBoard group (Mdn = 20.5) was significantly higher than their pre-test score (Mdn = 26.5), U = 56.5, p ¡ .001. In addition, the effect size (r = 0.78) was calculated as a rank-biserial correlation coefficient, suggesting the difference is large (r = 0.78). For the normal group, the post-test score (Mdn = 24) and pre-test score (Mdn= 21) also had significant difference (U = 105, p = .04). A medium effext size (r = 0.42) was found.

As for comparing the post-test scores of the two groups, the descriptive results suggested the HoloBoard group achieved a higher mean score (M = 26.39) in the post-test than the normal group (M = 24.61), although their mean pre-tests scores were rather similar (20.94 and 20.79, respectively). However, independent samples Mann-Whitney U test results indicated that the difference was not statically significant, U = 125.5, p = .13.

Table 3. Pre-/post-test Results (mean and standardized deviation) of Normal and HoloBoard group.

	HoloBoard (n=18)	Normal (n=18)	Effect size
Pre-test	20.10 (3.85)	20.78 (6.08 )	0.02
Post-test	26.40 (4.59)	24.61 (5.58)	0.19
Effect size	0.78 ***	0.42*

4.6.2. Student In-Class Engagement.

Students in HoloBoard group showed higher levels engagement in all the three dimensions: behavioral, emotional, and cognitive engagement according to our multimodal learning analytic results, which are shown in Table LABEL:tab:multimodalResults.

Table 4. Results of Student Engagement Analysis

Data	Variable	HoloBoard Group	Normal Group	Value	p
Video	Close Posture	Mdn = 40.5**	Mdn = 11.0	U = 257.5	0.002
	Neutral Posture	M = 38.28	M = 55.78*	t = 2.67	0.013
	Leave Posture	Mdn = 8.0	Mdn = 14	U = 156	0.864
	Positive Behavior	Mdn = 3.5*	Mdn = 0	U = 235.5	0.019
	Normal Behavior	Mdn = 82.0	Mdn = 77.5	U = 183.5	0.501
	Misbehavior	Mdn = 8.0	Mdn = 11.5	U = 125.5	0.252
	High Arousal/Positive Emotion	Mdn = 4.0*	Mdn = 0	U = 234.5	0.02
	High Arousal/Negative Emotion	Mdn = 0	Mdn = 0	U = 124.0	1
	Low Arousal/Positive Emotion	Mdn = 85.5	Mdn = 79.0	U = 193.0	0.339
	Low Arousal/Negative Emotion	Mdn = 0	Mdn = 11.0*	U = 82.5	0.011
Audio	Loudness	M = 0.000231***	M = 0.000153	t = 7.64	<.001
Audio	Frequency	M = 0.000032	M = 0.00017	t = -0.60	0.27
Transcripts	Speaking Turns/min	8.14	7.23	/	/
	Mean Sentence Length	M = 2.29	M = 2.35	t = 0.564	0.28
	Mean Speaking Turn Length	M = 2.42	M = 2.54	t = 0.840	0.2
	Student Speaking Times/min	M = 3.24***	M = 1.59	t = 26.155	<.001
	Words Spoken per Student/min	M = 7.86***	M = 4.10	t = -9.456	<.001
	Sentences Spoken per Student/min	M = 3.43***	M = 1.72	t = 20.892	<.001
	Assertion	Mdn = 3.05***	Mdn = 1.6	U = 308.0	<.001
	Questions	Mdn = 0.027**	Mdn = 0	U = 250.0	0.005
	Exclamations	Mdn = 0.08***	Mdn = 0.07	U = 324.0	<.001
	Remembering	Mdn = 1.54***	Mdn = 0.53	U = 316.0	<.001
	Understanding	Mdn = 0.46***	Mdn = 0.32	U = 279.0	<.001
	Applying	Mdn = 0	Mdn = 0	U = 124.0	0.239
	Analyzing	Mdn = 0.08	Mdn = 0.12	U = 116.0	0.152
	Evaluation	Mdn = 0	Mdn = 0	U = 144.0	0.584
	Creation	Mdn = 0	Mdn = 0	U = 170.0	0.815
	Positive Emotion	M = 1.36***	M = 1.20	t = -14.22	<.001

Behavioral Engagement Results Generally, students in the HoloBoard group were more engaged behaviorally than those in the normal group, indicated by more close posture and more positive classroom behavior. Mann-Whitney test indicated students in the HoloBoard group demonstrated significantly longer duration of close posture (p = .002) and positive behavior (p = .019) in the HoloBoard group than students in the normal group.

Emotional Engagement Results Students in the HoloBoard group exhibited higher levels of emotional engagement, which was demonstrated by more emotion that has high arousal and positive valance. This result was based on video and audio data coding, acoustic analysis, and semantic analysis.

For video coding results, the length of high arousal and positive emotion was significantly larger in the HoloBoard group (p = .02). We also found a significantly smaller length of low arousal and negative emotion in the HoloBoard group (p = .01). The audio data coding results also showed that students were more emotionally engaged in the HoloBoard class: the number of student exclamations per minute in the HoloBoard class was significantly larger than the one in the normal group (p ¡ .001). Moreover, t-tests were conducted to compare voice loudness and frequency of the HoloBoard and normal groups. There was a significant higher level of loudness in the HoloBoard group (p ¡ .001), whereas there was no significant difference in frequency between the two groups, indicated by the zero-crossing rate (p = .27). The greater voice loudness indicated higher levels of arousal in emotion of students in the HoloBoard group.

In addition, t-test was performed on the semantic data of the two groups’ transcripts. The results indicated there were significant more positive emotion (p ¡ .001) and probability of positivity (p ¡ .001) in the HoloBoard group than in the normal group. These results further confirmed the manual coding and acoustic analysis results, suggesting that the emotional engagement level of students in the HoloBoard group was higher.

Cognitive Engagement Results In general, students in the HoloBoard group showed more signs of cognitive engagement compared to those in the normal group. The signs included more contribution to classroom discussion as well as more responses in the remembering and understanding levels. Firstly, Mann-Whitney U tests were conducted on types of students’ speech. The results indicated that the numbers of students’ assertions (p ¡ .001) and questions (p = .005) per minute were both significantly larger in the HoloBoard group than the one in the normal group.

In addition, t-test results of comparing discourse attributes suggested the number of student speaking times in the HoloBoard group was significantly greater than the one in the normal group (p ¡ .001). In addition, the numbers of words and sentences students spoke per minute in HoloBoard group were also significantly larger than those in the normal group (both p-values are smaller than 0.001). There was no significant differences in other attributes. The results further confirmed the speech type analysis results, suggesting students in the HoloBoard group were more cognitively engaged and more willing to contribute to class discussion.

For the cognitive level of students’ responses, the numbers of remembering responses (p ¡ .001) and understanding responses (p ¡ .001) were significantly higher in the HoloBoard group than in the normal group. No significant differences were found in the number of responses at any higher cognitive level (e.g., applying and analyzing responses) between the two groups.

5. Discussion

5.1. Engaging Students with HoloBoard

The results of the experimental study show that HoloBoard can lead to significantly higher levels of engagement (i.e., behavioral, emotional, and cognitive) and slightly better learning outcomes in comparison to the normal class. Specifically, throughout the class, students in the HoloBoard group were found more active to participate in the class. The results indicated that students demonstrated more concentrating and positive behavior, higher levels of valance and arousal of emotion, more intensive rounds of discussion, and more remembering and understanding responses in the HoloBoard group than in the normal group. These signs of behavioral, emotional, and cognitive engagement observed in our study were consistent with the ones identified in previous literature, e.g., (Beşevli et al., 2019; Hsieh et al., 2015; Fredricks et al., 2004).

In particular, the techniques of the HoloBoard elicited more engaging activities from the students during the discussions. For instance, while studying the characteristics of planets with the role-play in the HoloBoard class, students consistently came up with ideas for the teacher to explore the space as an astronaut. Some of them, such as ”try to jump (on Mars),” and ”try to weigh yourself (on Mercury)” indicated the students’ intentions to apply their new knowledge of space (e.g., change of gravity) and their collective efforts to make meaning out of the information (Smart and Csapo, 2007; SRIVASTAVA, 2014; Davidson and Major, 2014; Matthews, 1996), which is the sign of students’ active learning (Smart and Csapo, 2007; SRIVASTAVA, 2014) and collaborative learning (Davidson and Major, 2014; Matthews, 1996). While using the virtual simulator demonstration to learn the order of planets, students were willing to share their opinions and get engaged in debates (e.g., where should the Earth be). They also exclaimed with excitement (e.g., ”Wow” and ”Yeah! We did it!”) when all the planets were placed in order. These anecdotes demonstrated that the usage of the HoloBoard created an engaging class where students could readily and actively participate and contribute.

In conclusion, with HoloBoard, the lecture-based class managed to create a more engaging atmosphere for active student participation.

5.2. Design Implications

5.2.1. Challenge of Role-play in Lecture-based Classes.

Although the augmented role-play function was welcomed by both teachers and students, there have raised other concerns. With our observation, proportion of teacher’s close-ended questions dramatically increased while using role-play, which caused the number of the students’ lower cognitive level in responses to increase. After simplifying the enter/exit action for role-play (i.e., we set a shortcut key link to this function from the original secondary menu.), the proportion of the experiment teacher’s close-ended questions returned to normal in the experiment study, and the experiment teacher’s self-report also confirmed the effectiveness of this change.

This suggested potential challenges when involving novel techniques such as role-play in lectures. On the one hand, unfamiliar technology increased teachers’ extraneous cognitive load and affected their teaching presence, as highlighted by previous research (Kerawalla et al., 2006). Therefore, the ease of use and necessary training are important factors in adopting novel lecturing techqniues On the other hand, role-play, as an effective teaching method, could be demanding for an instructor, both in preparation and in implementation (Ardriyati, 2009). Hence, when using role-play functions with HoloBoard in class, the system needs to support teachers (i.e., using AI assistants to help classroom management and learning process management) and allow them to control the pace of role-play easily.

5.2.2. Simulator Demonstration and Multi-user Game in Classroom.

We found that the simulator demonstration and the game brought particular excitement to the classroom. In the classroom, a very simple simulation demonstration of the planetary order not only attracted the attention of the students but also stimulated discussion on additional knowledge beyond the syllabus, for example, the orbital speed of the planets. Indeed, the experimenter teacher confirmed such positive effects and recalled that ”everyone was so excited when I finally put all the planets in the right order.” She was impressed by the visual and intuitive way of displaying content materials and believed that ”such direct observation and perception of orders and orbital speed of the planets can have amazing effects.”

In addition, the multi-user game, enabled by the large-format screen of the HoloBoard, also resulted in more learning opportunities for students. In the post-test self-reports, the students in the HoloBoard group had a significantly higher collaborative-learning-related self-evaluation score. They also expressed that they felt their teams had more trust in other members, and they had more frequently encouraged other members to join the group activities. The experimenter teacher also thought ”the game allowed students to apply the knowledge they just learned.”

In addition, we agree that the interactive designs of HoloBoard should be able to sustain student engagement in the long run. Although our current study only compared immediate influence, we believe with the simulator demonstration of content knowledge and gamification elements discussed above, have the potential to overcome the novelty effect. because we have embedded pedagogical factors in our design, for example, selecting the appropriate knowledge to stimulate and presenting the learning journey mading up of raising questions, exploration, and end game. Such ways to make the engagement meaningful and helpful for students are suggested ways to overcome the novelty effects in (Tsay et al., 2020).

In summary, simulator demonstrations and games enabled with HoloBoard can lead to positive student experiences in multi-user learning environments and we recommend including more such activities in classrooms.

5.2.3. Interactive Transparent Screen for Children

We offered a free-play session where students could play the virtual multi-user game on the HoloBoard after the class. We observed that the students made good use of the double-sided feature of the HoloBoard and engaged in cooperative pretend play on the two opposite sides of the HoloBoard spontaneously. For example, in one of the pretend play scenarios, some students in front of the HoloBoard waved controllers in the air and create the visual and audio effects reflecting the hitting of stones, pretending there were attacking students behind the HoloBoard. Those students behind the HoloBoard traced the location of the effects and pretended that they got hurt and fell down.

Pretend play has long been recognized as a vital type of children’s play from the perspective of children’s cognitive, social, and physical development (Vygotsky, 1980; Lillard, 1993). Furthermore, children who engaged in more pretend play generally have more advanced social-cognitive skills, including social perspective taking (Fein, 1981) and social competence (Connolly and Doyle, 1984). Hence, the development of children’ higher-level cognition is facilitated (Bergen, 2002).

HoloBoard showed strong potential as a platform for active and beneficial interaction between children. Beyond classrooms, such interactive transparent displays could be installed at children’s playgrounds to facilitate more pretend play and social play via the double-sided interaction feature.

5.3. Limitation and Future work

For the HoloBoard system, regarding the expectation of potential implementation in the classroom settings, there are several concerns such as a dimmer display color compared to ordinary LED displays, the requirement of trackers and controllers, and so on. Nevertheless, most participants expressed their expectations to see HoloBoard in actual classrooms in the near future. In addition, since the hardware cost and maintenance of HoloBoard is manageable, we will focus on optimizing creative tools like Chalktalk (Perlin et al., 2018) and body-driven augmented graphics creative tool (Saquib et al., 2019) to make it feasible for teachers to independently develop content like slides or role-playing in the future.

For the study, the current experimental study did not fully utilized the functions of HoloBoard. This was for several reasons. First, it was non-trivial to design a course that includes all the novel functions of the system. Interaction techniques such as ”augmented dashboard”, ”physical to virtual interactions”, ”3D modeling”, and ”holographic telepresence” shall be designed within other course contents and lecturing contexts. Second, it is fairly time-consuming for the experimental teachers to understand and get used to the new functions. Therefore, how to best integrate these techniques of the HoloBoard systems into a lecture requires further explorations. Moreover, we acknowledge that the current study design comparing student interaction during the two classes can not help us estimate possible novelty effects. Verifying and reducing possible novelty effects in long-term usage will be our future work.

Meanwhile, due to individual differences among learners and novelty effects of new technology, it remains a challenge to exploit the advantages of such immersive technology in wild. In the future, we plan to leverage HoloBoard in long-term use cases and expand on a larger scale.

6. Conclusions

We presented HoloBoard, a large-format immersive teaching board based on pseudo holographics. Its unique features of immersive visual presentation and transparent screen allowed us to explore a rich set of novel interaction techniques, potentially valuable for lecturing in classroom. To verify the effect of HoloBoard in actual classes, we designed a course and carried out a comparative study with 36 primary school students. The multimodal learning analytics results of the experiment demonstrated that students in the HoloBoard group were more engaged behaviorally, emotionally, and cognitively, indicating that immersive technology can benefit learning in lecture-based classes. We expect this work can contribute to the educational technology research community by providing a novel interactive system and a deep understanding of the immersive technology to support learning.

Acknowledgements.

This work was supported by the China National Key Research and Development Plan under Grant No. 2020YFF0305405. Our sincere thanks to all the participants of our experiment.

References

(1)
Adachi et al. (2013) Takayuki Adachi, Masafumi Goseki, Keita Muratsu, Hiroshi Mizoguchi, Miki Namatame, Masanori Sugimoto, Fusako Kusunoki, Etsuji Yamaguchi, Shigenori Inagaki, and Yoshiaki Takeda. 2013. Human SUGOROKU: Full-Body Interaction System for Students to Learn Vegetation Succession (IDC ’13). Association for Computing Machinery, New York, NY, USA, 4 pages. https://doi.org/10.1145/2485760.2485830
Alyüz et al. (2016) Nese Alyüz, Eda Okur, Ece Oktay, Utku Genc, Sinem Aslan, Sinem Emine Mete, David Stanhill, Bert Arnrich, and Asli Arslan Esme. 2016. Towards an emotional engagement model: Can affective states of a learner be automatically detected in a 1: 1 learning scenario?. In UMAP (Extended Proceedings).
Aman et al. (2016) Abdulrazak M Aman, Houcine Meddour, Abdul HA Majid, and Mohamed AA Auf. 2016. Exploring the use of holographic telepresence in designing virtual learning environments: A Saudi experience. Journal of Economic & Management Perspectives 10, 4 (2016), 610–621.
Apostolellis and Bowman (2014) Panagiotis Apostolellis and Doug A Bowman. 2014. Evaluating the effects of orchestrated, game-based learning in virtual environments for informal education. In Proceedings of the 11th Conference on Advances in Computer Entertainment Technology. 1–10.
Ardriyati (2009) Wienny Ardriyati. 2009. ROLEPLAY: ONE ALTERNATIVE AND EFFETIVE TEACHING METHOD TO IMPROVE STUDENTS’COMMUNICATION SKILL. Dinamika Bahasa dan Budaya 3, 2 (2009), 218–228.
Aslan et al. (2019) Sinem Aslan, Nese Alyuz, Cagri Tanriover, Sinem E Mete, Eda Okur, Sidney K D’Mello, and Asli Arslan Esme. 2019. Investigating the impact of a real-time, multimodal student engagement analytics technology in authentic classrooms. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
Balta and Duran (2015) Nuri Balta and Muharrem Duran. 2015. Attitudes of students and teachers towards the use of interactive whiteboards in elementary and secondary school classrooms. Turkish Online Journal of Educational Technology-TOJET 14, 2 (2015), 15–21.
Bateman et al. (2017) John Bateman, Janina Wildfeuer, and Tuomo Hiippala. 2017. Multimodality: Foundations, research and analysis–A problem-oriented introduction. Walter de Gruyter GmbH & Co KG.
Bergen (2002) Doris Bergen. 2002. The role of pretend play in children’s cognitive development. Early Childhood Research & Practice 4, 1 (2002), n1.
Beşevli et al. (2019) Ceylan Beşevli, Elif Salman, Tilbe Goksun, Hakan Urey, and Oğuzhan Özcan. 2019. MaR-T: Designing a projection-based mixed reality system for nonsymbolic math development of preschoolers: Guided by theories of cognition and learning. In Proceedings of the 18th ACM International Conference on Interaction Design and Children. 280–292.
Betcher and Lee (2009) Chris Betcher and Mal Lee. 2009. The interactive whiteboard revolution: Teaching with IWBs. Aust Council for Ed Research.
Black et al. (2009) Matthew Black, Jeannette Chang, Jonathan Chang, and Shrikanth Narayanan. 2009. Comparison of child-human and child-computer interactions based on manual annotations. In Proceedings of the 2nd Workshop on Child, Computer and Interaction. 1–6.
Bloom et al. (1956) Benjamin S Bloom et al. 1956. Taxonomy of educational objectives. Vol. 1: Cognitive domain. New York: McKay 20 (1956), 24.
Charlton (2006) Bruce G Charlton. 2006. Lectures are such an effective teaching method because they exploit evolved human psychology to improve learning.
Connolly and Doyle (1984) Jennifer A Connolly and Anna-Beth Doyle. 1984. Relation of social fantasy play to social competence in preschoolers. Developmental Psychology 20, 5 (1984), 797.
Corbin and Strauss (2014) Juliet Corbin and Anselm Strauss. 2014. Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage publications.
Couse and Chen (2010) Leslie J Couse and Dora W Chen. 2010. A tablet computer for young children? Exploring its viability for early childhood education. Journal of research on technology in education 43, 1 (2010), 75–96.
Davidson and Major (2014) Neil Davidson and Claire Howell Major. 2014. Boundary crossings: Cooperative learning, collaborative learning, and problem-based learning. Journal on excellence in college teaching 25 (2014).
Di Lascio et al. (2018) Elena Di Lascio, Shkurta Gashi, and Silvia Santini. 2018. Unobtrusive assessment of students’ emotional engagement during lectures using electrodermal activity sensors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1–21.
Duran and Duran (2004) Lena Ballone Duran and Emilio Duran. 2004. The 5E Instructional Model: A Learning Cycle Approach for Inquiry-Based Science Teaching. Science Education Review 3, 2 (2004), 49–58.
Enyedy et al. (2012) Noel Enyedy, Joshua A Danish, Girlie Delacruz, and Melissa Kumar. 2012. Learning physics through play in an augmented reality environment. International journal of computer-supported collaborative learning 7, 3 (2012), 347–378.
Fein (1981) Greta G Fein. 1981. Pretend play in childhood: An integrative review. Child development (1981), 1095–1118.
Ferracani et al. (2014) Andrea Ferracani, Daniele Pezzatini, and Alberto Del Bimbo. 2014. A natural and immersive virtual interface for the surgical safety checklist training. In Proceedings of the 2014 ACM international workshop on serious games. 27–32.
Fjeld et al. (2007) Morten Fjeld, Jonas Fredriksson, Martin Ejdestig, Florin Duca, Kristina Bötschi, Benedikt Voegtli, and Patrick Juchli. 2007. Tangible user interface for chemistry education: comparative evaluation and re-design. In Proceedings of the SIGCHI conference on Human factors in computing systems. 805–808.
Flanders (1970) Ned A Flanders. 1970. Analyzing teaching behavior. (1970).
Fredricks et al. (2004) Jennifer A Fredricks, Phyllis C Blumenfeld, and Alison H Paris. 2004. School engagement: Potential of the concept, state of the evidence. Review of educational research 74, 1 (2004), 59–109.
Freina and Ott (2015) Laura Freina and Michela Ott. 2015. A literature review on immersive virtual reality in education: state of the art and perspectives. In The international scientific conference elearning and software for education, Vol. 1. 10–1007.
Furió et al. (2013) David Furió, Santiago GonzáLez-Gancedo, M-Carmen Juan, Ignacio Seguí, and MaríA Costa. 2013. The effects of the size and weight of a mobile device on an educational game. Computers & Education 64 (2013), 24–41.
Gallagher and Lindgren (2015) Shaun Gallagher and Robb Lindgren. 2015. Enactive metaphors: Learning through full-body engagement. Educational Psychology Review 27, 3 (2015), 391–404.
Gao et al. (2020) Nan Gao, Wei Shao, Mohammad Saiedur Rahaman, and Flora D Salim. 2020. n-Gage: Predicting in-class Emotional, Behavioural and Cognitive Engagement in the Wild. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3 (2020), 1–26.
Gelsomini et al. (2020) Mirko Gelsomini, Giulia Leonardi, and Franca Garzotto. 2020. Embodied Learning in Immersive Smart Spaces. Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376667
Gran (2018) Marius Wahl Gran. 2018. Tabula Rasa-The performative and unique materiality of the blackboard. RoSE–Research on Steiner Education 9, 1 (2018).
Groccia (2018) James E Groccia. 2018. What is student engagement? New Directions for Teaching and Learning 2018, 154 (2018), 11–20.
Han et al. (2014) Jaehyun Han, Seongkook Heo, Jiseong Gu, and Geehyuk Lee. 2014. Trampoline: a double-sided elastic touch device for repoussé and chasing techniques. In CHI’14 Extended Abstracts on Human Factors in Computing Systems. 1627–1632.
Haßler et al. (2016) Bjoern Haßler, Louis Major, and Sara Hennessy. 2016. Tablet use in schools: A critical review of the evidence for learning outcomes. Journal of Computer Assisted Learning 32, 2 (2016), 139–156.
Holstein et al. (2018) Kenneth Holstein, Gena Hong, Mera Tegene, Bruce M McLaren, and Vincent Aleven. 2018. The classroom as a dashboard: co-designing wearable cognitive augmentation for K-12 teachers. In Proceedings of the 8th international conference on learning Analytics and knowledge. 79–88.
Homer et al. (2018) Ryan Homer, Khe Foon Hew, and Cheng Yong Tan. 2018. Comparing digital badges-and-points with classroom token systems: Effects on elementary school ESL students’ classroom behavior and English learning. Journal of Educational Technology & Society 21, 1 (2018), 137–151.
Hsieh et al. (2015) Ya-Hui Hsieh, Lin Yi-Chun, and Huei-Tse Hou. 2015. Exploring elementary-school students’ engagement patterns in a game-based learning environment. Journal of Educational Technology & Society 18, 2 (2015), 336.
Huang et al. (2010) Hsiu-Mei Huang, Ulrich Rauch, and Shu-Sheng Liaw. 2010. Investigating learners’ attitudes toward virtual reality learning environments: Based on a constructivist approach. Computers & Education 55, 3 (2010), 1171–1182.
Huitt and Hummel (2003) William Huitt and John Hummel. 2003. Piaget’s theory of cognitive development. Educational psychology interactive 3, 2 (2003), 1–5.
Ibáñez et al. (2014) María Blanca Ibáñez, Ángela Di Serio, Diego Villarán, and Carlos Delgado Kloos. 2014. Experimenting with electromagnetism using augmented reality: Impact on flow student experience and educational effectiveness. Computers & Education 71 (2014), 1–13.
Ibrahim et al. (2018) Adam Ibrahim, Brandon Huynh, Jonathan Downey, Tobias Höllerer, Dorothy Chun, and John O’donovan. 2018. Arbis pictus: A study of vocabulary learning with augmented reality. IEEE transactions on visualization and computer graphics 24, 11 (2018), 2867–2874.
Ishii and Kobayashi (1992) Hiroshi Ishii and Minoru Kobayashi. 1992. Clearboard: A seamless medium for shared drawing and conversation with eye contact. In Proceedings of the SIGCHI conference on Human factors in computing systems. 525–532.
Jones et al. (2014) Brett Jones, Rajinder Sodhi, Michael Murdock, Ravish Mehra, Hrvoje Benko, Andrew Wilson, Eyal Ofek, Blair MacIntyre, Nikunj Raghuvanshi, and Lior Shapira. 2014. Roomalive: Magical experiences enabled by scalable, adaptive projector-camera units. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 637–644.
Jones et al. (2013) Brett R Jones, Hrvoje Benko, Eyal Ofek, and Andrew D Wilson. 2013. IllumiRoom: peripheral projected illusions for interactive experiences. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 869–878.
Kang et al. (2016) Seokbin Kang, Leyla Norooz, Vanessa Oguamanam, Angelisa C. Plane, Tamara L. Clegg, and Jon E. Froehlich. 2016. SharedPhys: Live Physiological Sensing, Whole-Body Interaction, and Large-Screen Visualizations to Support Shared Inquiry Experiences. In Proceedings of the The 15th International Conference on Interaction Design and Children (Manchester, United Kingdom) (IDC ’16). Association for Computing Machinery, New York, NY, USA, 275–287. https://doi.org/10.1145/2930674.2930710
Karsenti (2016) Thierry Karsenti. 2016. The interactive whiteboard: Uses, benefits, and challenges. A survey of 11,683 students and 1,131 teachers. Canadian Journal of Learning and Technology/La revue canadienne de l’apprentissage et de la technologie 42, 5 (2016).
Kerawalla et al. (2006) Lucinda Kerawalla, Rosemary Luckin, Simon Seljeflot, and Adrian Woolard. 2006. “Making it real”: exploring the potential of augmented reality for teaching primary school science. Virtual reality 10, 3-4 (2006), 163–174.
Kontra et al. (2015) Carly Kontra, Daniel J Lyons, Susan M Fischer, and Sian L Beilock. 2015. Physical experience enhances science learning. Psychological science 26, 6 (2015), 737–749.
Lee and Kinzie (2012) Youngju Lee and Mable B Kinzie. 2012. Teacher question and student response with regard to cognition and language use. Instructional Science 40, 6 (2012), 857–874.
Lillard (1993) Angeline S Lillard. 1993. Pretend play skills and the child’s theory of mind. Child development 64, 2 (1993), 348–371.
Lindgren and Johnson-Glenberg (2013) Robb Lindgren and Mina Johnson-Glenberg. 2013. Emboldened by embodiment: Six precepts for research on embodied learning and mixed reality. Educational researcher 42, 8 (2013), 445–452.
Luévano et al. (2015) Eduardo Luévano, Eduardo López de Lara, and Juan Edward Castro. 2015. Use of telepresence and holographic projection mobile device for college degree level. Procedia Computer Science 75 (2015), 339–347.
Lui (2018) Michelle Lui. 2018. Designing for student interactions: The role of embodied interactions in mediating collective inquiry in an immersive simulation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.
Matthews (1996) Roberta S Matthews. 1996. Collaborative learning: Creating knowledge with students. Teaching on solid ground: Using scholarship to improve practice 1996 (1996), 101–124.
Mikropoulos and Natsis (2011) Tassos A Mikropoulos and Antonis Natsis. 2011. Educational virtual environments: A ten-year review of empirical research (1999–2009). Computers & Education 56, 3 (2011), 769–780.
Morgan (2008) Georgia Lyn Morgan. 2008. Improving student engagement: Use of the interactive whiteboard as an instructional tool to improve engagement and behavior in the junior high school classroom. (2008).
Mulder et al. (2002) Ingrid Mulder, Janine Swaak, and Joseph Kessels. 2002. Assessing group learning and shared understanding in technology-mediated interaction. Journal of Educational Technology & Society 5, 1 (2002), 35–47.
Müller et al. (2014) Mathias Müller, Anja Knöfel, Thomas Gründer, Ingmar Franke, and Rainer Groh. 2014. Flexiwall: Exploring layered data with elastic displays. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces. 439–442.
Murcia and Sheffield (2010) Karen Murcia and R. Sheffield. 2010. Talking about science in interactive whiteboard classrooms. Australasian Journal of Educational Technology 26 (2010), 417–431.
Muttappallymyalil et al. (2016) Jayakumary Muttappallymyalil, Susirith Mendis, Lisha Jenny John, Nisha Shanthakumari, Jayadevan Sreedharan, and Rizwana B Shaikh. 2016. Evolution of technology in teaching: Blackboard and beyond in Medical Education. Nepal journal of epidemiology 6, 3 (2016), 588.
Nedungadi et al. (2013) Prema Nedungadi, Raghu Raman, and Mark McGregor. 2013. Enhanced STEM learning with Online Labs: Empirical study comparing physical labs, tablets and desktops. In 2013 IEEE Frontiers in Education conference (FIE). IEEE, 1585–1590.
Oviatt (2018) Sharon Oviatt. 2018. Ten Opportunities and challenges for advancing student-centered multimodal learning analytics. In Proceedings of the 20th ACM International Conference on Multimodal Interaction. 87–94.
Özgür et al. (2017) Ayberk Özgür, Séverin Lemaignan, Wafa Johal, Maria Beltran, Manon Briod, Léa Pereyre, Francesco Mondada, and Pierre Dillenbourg. 2017. Cellulo: Versatile handheld robots for education. In 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI. IEEE, 119–127.
Paredes and Vázquez (2019) Sandra Gudino Paredes and Nohemí Rivera Vázquez. 2019. My Teacher is a Hologram: Measuring innovative STEM learning experiences. In 2019 IEEE Integrated STEM Education Conference (ISEC). IEEE, 332–337.
Pas et al. (2015) Elise T Pas, Anne H Cash, Lindsey O’Brennan, Katrina J Debnam, and Catherine P Bradshaw. 2015. Profiles of classroom behavior in high schools: Associations with teacher behavior management strategies and classroom composition. Journal of School Psychology 53, 2 (2015), 137–148.
Pecher and Zwaan (2005) Diane Pecher and Rolf A Zwaan. 2005. Grounding cognition: The role of perception and action in memory, language, and thinking. Cambridge University Press.
Penuel et al. (2007) William R Penuel, Jeremy Roschelle, and Nicole Shechtman. 2007. Designing formative assessment software with teachers: An analysis of the co-design process. Research and practice in technology enhanced learning 2, 01 (2007), 51–74.
Perlin et al. (2018) Ken Perlin, Zhenyi He, and Fengyuan Zhu. 2018. Chalktalk vr/ar. International SERIES on Information Systems and Management in Creative eMedia (CreMedia) 2017/2 (2018), 30–31.
Radu (2014) Iulian Radu. 2014. Augmented reality in education: a meta-review and cross-media analysis. Personal and Ubiquitous Computing 18, 6 (2014), 1533–1543.
Rizzo et al. (2006) Albert A Rizzo, Todd Bowerly, J Galen Buckwalter, Dean Klimchuk, Roman Mitura, and Thomas D Parsons. 2006. A virtual reality scenario for all seasons: the virtual classroom. Cns Spectrums 11, 1 (2006), 35–44.
Saquib et al. (2019) Nazmus Saquib, Rubaiat Habib Kazi, Li-Yi Wei, and Wilmot Li. 2019. Interactive body-driven graphics for augmented video performance. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
Schneider et al. (2016) Bertrand Schneider, Kshitij Sharma, Sébastien Cuendet, Guillaume Zufferey, Pierre Dillenbourg, and Roy Pea. 2016. Using Mobile Eye-Trackers to Unpack the Perceptual Benefits of a Tangible User Interface for Collaborative Learning. 23, 6, Article 39 (Dec. 2016), 23 pages. https://doi.org/10.1145/3012009
Sharma et al. (2013) Sharad Sharma, Ruth Agada, and Jeff Ruffin. 2013. Virtual reality classroom as an constructivist approach. In 2013 Proceedings of IEEE Southeastcon. IEEE, 1–5.
Smart and Marshall (2013) Julie B Smart and Jeff C Marshall. 2013. Interactions between classroom discourse, teacher questioning, and student cognitive engagement in middle school science. Journal of Science Teacher Education 24, 2 (2013), 249–267.
Smart and Csapo (2007) Karl L Smart and Nancy Csapo. 2007. Learning by doing: Engaging students through learner-centered activities. Business Communication Quarterly 70, 4 (2007), 451–457.
Song et al. (2011) Kyohyun Song, Gunhee Kim, Inkyu Han, Jeongyoung Lee, Ji-Hyung Park, and Sungdo Ha. 2011. CheMO: mixed object instruments and interactions for tangible chemistry experiments. In CHI’11 Extended Abstracts on Human Factors in Computing Systems. 2305–2310.
SRIVASTAVA (2014) KIRAN SRIVASTAVA. 2014. Role of flipped classroom in education. Education 3, 4 (2014).
Stroud (2015) Robert Stroud. 2015. Learner Classroom Engagement: Definition, Measurement and Data Usage. In Asian Conference on Psychology and the Behavioral Sciences Official Conference Proceedings. 113–125.
Tanaka and Matsuzoe (2012) Fumihide Tanaka and Shizuko Matsuzoe. 2012. Children teach a care-receiving robot to promote their learning: Field experiments in a classroom for vocabulary learning. Journal of Human-Robot Interaction 1, 1 (2012), 78–95.
Tian et al. (2020) Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, and Feng Wu. 2020. SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis. arXiv preprint arXiv:2005.05635 (2020).
Trees and Jackson (2007) April R. Trees and M. Jackson. 2007. The learning environment in clicker classrooms: student processes of learning and involvement in large university‐level courses using student response systems. Learning, Media and Technology 32 (2007), 21 – 40.
Tsay et al. (2020) Crystal Han-Huei Tsay, Alexander K Kofinas, Smita K Trivedi, and Yang Yang. 2020. Overcoming the novelty effect in online gamified learning systems: An empirical evaluation of student engagement and performance. Journal of Computer Assisted Learning 36, 2 (2020), 128–146.
Vazquez et al. (2017) Christian David Vazquez, Afika Ayanda Nyati, Alexander Luh, Megan Fu, Takako Aikawa, and Pattie Maes. 2017. Serendipitous language learning in mixed reality. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 2172–2179.
Vygotsky (1980) Lev Semenovich Vygotsky. 1980. Mind in society: The development of higher psychological processes. Harvard university press.
Wagner et al. (2005) Johannes Wagner, Jonghwa Kim, and Elisabeth André. 2005. From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification. In 2005 IEEE international conference on multimedia and expo. IEEE, 940–943.
Watanabe et al. (2008) Yoshihiro Watanabe, Alvaro Cassinelli, Takashi Komuro, and Masatoshi Ishikawa. 2008. The deformable workspace: A membrane between real and virtual space. In 2008 3rd IEEE International Workshop on Horizontal Interactive Human Computer Systems. IEEE, 145–152.
Worsley and Blikstein (2015) Marcelo Worsley and Paulo Blikstein. 2015. Leveraging multimodal learning analytics to differentiate student learning strategies. In Proceedings of the Fifth international conference on learning analytics and knowledge. 360–367.
Xie et al. (2008) Lesley Xie, Alissa N Antle, and Nima Motamedi. 2008. Are tangibles more fun? Comparing children’s enjoyment and engagement using physical, graphical and tangible user interfaces. In Proceedings of the 2nd international conference on Tangible and embedded interaction. 191–198.
Yang et al. (2012) Kai-Ti Yang, Tzu-Hua Wang, and Ying-Chen Kao. 2012. How an interactive whiteboard impacts a traditional classroom. Education as Change 16, 2 (2012), 313–332.
Zhu (2006) Erping Zhu. 2006. Interaction and cognitive engagement: An analysis of four asynchronous online discussions. Instructional Science 34, 6 (2006), 451.
Șerban et al. (2017) Ovidiu Șerban, Mukesh Barange, Sahba Zojaji, Alexandre Pauchet, Adeline Richard, and Emilie Chanoni. 2017. Interactive narration with a child: impact of prosody and facial expressions. In Proceedings of the 19th ACM International Conference on Multimodal Interaction. 23–31.

Appendix A Pre-/Post- Knowledge Test

A.1. Part One: Open-ended Questions

(1)

Do you know any story about human exploring the space? You can tell me as detailed as possible.
(2)

Do you think why human explore the space? You can tell me as much as possible about the reasons that human explore the space.
(3)

Do you think why human explore the space? You can tell me as much as possible about the reasons that human explore the space.
(4)

Please describe the solar system.
(5)

Can you tell me what are the planets in the solar system? How can they be categorized? What are the characteristics of each of them?”
(6)

Do you think human can live freely in other planets? Why? What are the needed elements to make a planet habitable?

A.2. Part Two: Singe-choice Questions

(1)

What is the largest planet in the solar system?

a. Earth b. Jupiter c. Sun d. Uranus
(2)

In the solar system, what is the farthest planets from the sun?

a. Earth b. Saturn c. Mars d. Neptune
(3)

What is the hottest planet in the solar system?

a. Venus b. Saturn c. Mars d. Jupiter
(4)

What is the heaviest planet in the solar system? (the heaviest; the highest in mass)

a. Mercury b. Earth c. Mars d. Jupiter

A.3. Part Three: Multiple-choice Questions

(1)

In the following planets, what is/are the gas planet(s)?

a. Mercury b. Venus c. Neptune d. Mars
(2)

What is/are the factor(s) effecting human moving to and living in other planet(s)?

a. Water b. Oxygen c. Temperature d. Distance