Can Social Robots Effectively Elicit Curiosity in STEM Topics from K-1 Students During Oral Assessments?

A sample of 164 kindergarten and first-grade students, consisting of 53 kindergarteners and 111 first graders from a Southern California elementary school, were recruited for the study. Selection criteria included parental consent and completion of all tasks in the study. All of the students were English speaking with some reporting additional language exposure (most often Spanish) at home.

Interactions with each student were recorded with a Logitech C920 Webcam microphone with a sampling rate of 48kHz. Each student was approximately two feet from the test administrator (either JIBO or the human assessor), and the microphone was placed equidistantly between them. Recordings took place in an empty office at the school site during school hours to simulate a realistic environment in which JIBO could be used.

Data collection followed the protocol described in ([12, 13, 14]). The children's speech samples were collected using Goldman-Fristoe Test of Articulation-3 (GFTA-3) [15] sounds in sentences protocol. Each student was first read a story appropriate for their grade level. The kindergarten students were read a story containing 20 simple sentences and the first-grade students were read a story containing 15 sentences mixing both simple and compound structures. As a fictitious example (the actual assessment material is under copyright restrictions), a sentence such as, “The cat was fat, fuzzy, and orange,” may have appeared in the story for first-graders. After the story was read once in its entirety, the story was repeated sentence by sentence, and the student was asked to repeat each sentence back to the test administerer. Ten kindergarten students and ten first-grade students were administered the test by a human assessor, and the others were administered the test by JIBO. Both gave the same prompts to the children. In giving each prompt, JIBO played a corresponding audio file containing a recording of an adult woman whose voice was pitch shifted up to match the pitch of a child and slowed down. For sessions in which the child was given the prompts by JIBO, a human assessor was present and intervened if the child was unable to complete the task with only the social robot's instructions. In sessions in which the child was given the prompt by a human assessor, they gave the prompt to the child a second time if they were unable to repeat the sentence after hearing it once.

Any questions or comments that the child made referring to JIBO before, during, or after the assessment were documented and categorized by their different characteristics.

The 10 kindergarteners and 10 first-graders who completed the assessment with only the human assessor were compared to a group of 10 kindergarteners and 10 first-graders who completed the same assessment administered by the social robot. The robot-assessed children in the comparison group were selected from the total participant pool on the basis that they were assessed in the same time period and overseen by same human assessor as the students in the human-assessed children. The following metrics were calculated for comparison between the two conditions:

1. 1.

   Accuracy: each sentence that the students were asked to repeat was scored out of the total number of words in the sentence. One point was given for each word that the child repeated correctly in the correct order.

2. 2.

   Need for additional prompting: we noted each time that a child required additional prompting to repeat a sentence and noted the reason for the additional prompting. If the child was silent for at least 3 seconds after given the prompt, the reason was noted as “reticence.” If the child forgot the prompt or did not understand what to say, the reason is given as “needs repetition.” If the child said something unrelated to the prompt then the reason was listed as “distracted.”

3. 3.

   Pitch correlation: An additional question is whether students attempt to follow the administrator’s pitch and rhythm during the assessment. While not directly related to speech production assessment quality, differences in how the children incorporate the human and the robot’s prosody into their repetition of the prompt may indicate how engaged the children were with the assessor. For each prompt, the pitch contour of both the administerer (JIBO or the human assessor) uttering the prompt and the child's voice while repeating it were extracted using Praat's [16] pitch tracking algorithm with parameters set to best measure a child's pitch. The Praat pitch contours were manually corrected if any errors occurred in the pitch tracking algorithm. Then the Pearson correlation coefficient between the child's pitch contour and administrator's (either the robot or human assessor) pitch contour was calculated and taken as a measure of the perceptual similarity of the two as in [17].

Sentence word accuracy scores were scored manually by the researchers who are experienced in evaluating children's speech. Reasons for the child needing additional prompting were also noted manually following the above criteria.

The students were allowed to interact with the robot and speak freely before, during, and after the assessment. Any questions or comments made by students about JIBO that may be used to grow interest in robotics or STEM were characterized by the following five categories:

1. 1.

   Mechanical: Referring to how robots work or asking about fundamentals of robotics or other STEM concepts

2. 2.

   Functional: Referring to why JIBO looks and acts as it does or asking about engineering design choices

3. 3.

   Relational: Relating JIBO to something that the student has seen previously

4. 4.

   Personifying: Assigning human characteristics to JIBO

5. 5.

   Hypothetical: Exploratory questions about JIBO or related STEM concepts

Questions/comments could by characterized by more than one category. Some examples of children’s comments are given in Table I.

Table II shows the percent word accuracy of the repetition task for children in each group. The maximum score, minimum score, and standard deviation for the group are also given.

Table III gives the number of times students in the two groups required additional prompting to be able to complete the exercise and the reason for the need for additional prompting.

Average correlation between the assessor and students’ pitch contours when uttering the same prompt are shown in Table IV for each group. The percentage of repetitions spoken by students that are significantly correlated in pitch contour to the prompt given by the assessor are also given in this Table.

In total, 43 kindergarteners and 101 first-graders were assessed using JIBO. Table V shows the percentage of these children in each grade level who made a comment or question in the given category. In total, approximately 45% of the students made comments or questions about JIBO pertaining to at least one of the categories.

As shown in Table II, in the repetition task, the students' performance was not significantly affected by the presence of the social robot, JIBO. The table shows that when repeating words to the examiner, neither the kindergarten students ($p>0.1$) nor the first-grade students ($p>0.05$) are significantly worse at repeating words with JIBO than they are with a human. Although the mean sentence score was about 2% higher when the test was administered by a human assessor for both the kindergarten and first-grade students, this number is not statistically significant ($p>0.05$). There is however a significantly larger standard deviation in the sessions administered by JIBO (note the differences between the minimum and maximum sentence scores).

Table III shows that kindergarteners are more reticent when the test is administered by JIBO than by a human assessor. However, this reticence typically only occurs at the beginning of the session and disappears as the students are exposed more to the social robot. A possible explanation for this finding is that at this early stage of the activity, students may have been adapting to the activity. This trend in reticence does not extend to the first-grade students. Most of the first-grade students did need sentences to be repeated more often by the human facilitator present when the test was administered by JIBO. It is worth noting that the first-grade student who needed intervention most often when using JIBO predominately spoke a language other than English at home. The prompts that needed repetition were most often those containing longer sentences with compound structures.

Table IV implies that the pitch contours of the students' repetition of the prompt more closely match those of the prompt given by the human assessor. This means that while repeating sentences said by the assessor, the students’ changes in pitch and rhythm matched the human assessor more closely than the robot. This may be due to the robot having less natural sounding prosody, making it more difficult to match. However, these correlations are not significant, and we would need a larger sample size to definitively state that the use of a robot affects whether or not the students follow the speech prosody of the human assessor more. It is important to note that the correlations for first graders are lower than those of the kindergarteners. This is likely due to the fact that the prompts read to the first graders were longer and contained more complex sentence structures, requiring the students to turn their attention more towards remembering the words than matching the prosody given.

Almost half of the students made inquiries as to how robots work, why they are designed in the way that they are, and other ideas that are conducive to furthering children’s interest in STEM. The kindergarteners most commonly asked questions related to how JIBO is able to move, speak, and show images. That is, their questions typically asked for explanations to observed phenomenon. This implies that these interactions can be used to motivate kindergarten-level lessons on more visual concepts in robotics and STEM like motors, cameras, and programming fundamentals. The first-graders also asked a large number of questions of this nature, albeit fewer on average than the kindergarteners, while also asking personifying and hypothetical questions that delve past just what they can directly observe in the robot, such as, “Where does JIBO sleep?” and “Can you make him do [a different type of movement]?” These questions may be used to motivate further interest into engineering concepts such as biological inspiration for design and user-centered design. We also note that only the first graders related JIBO to other robots that they had seen previously. This may be because the younger students have had less exposure to other instances of robots.

We observed that students became excited to participate in the assessment when they learned that they would be working with a robot. This experience was, for most, if not all, of the students, their first interaction with a social robot. From the students’ excitement and frequency of questions asked, we believe that JIBO can be used to prime young students for robotic and STEM-centered lessons, possibly motivating them to further pursue them in secondary and post-secondary education. We believe that many of the students who did not make comments or questions about JIBO were simply shy, as our work shows that even the students working with only the human assessor still displayed a large amount of reticence throughout the assessment. Further interactions with the robot to build familiarity may reduce that reticence over time and lead to additional questions from students in the future.