Let’s be friends! A rapport-building 3D embodied conversational agent for the Human Support Robot

Forthcoming human-support robots are anticipated to assist people in a variety of contexts (Dahl and Boulos, 2014; Šabanović, 2010) involving socio-emotional personal information (e.g. helping an elderly person live safely independently) best communicated to humans via their innate communication modalities (e.g. speech, facial expressions, gestures), and ideally with established rapport between interlocutors. We aim to continue investigating the introduction of ECAs on social and service robots as a natural user interface metaphor for HRI (Peña et al., 2018; Goodrich and Schultz, 2008).

Establishing and maintaining rapport between humans is a proven determinant of positive communication outcomes, and is the result of a combination of highly socio-cultural-emotional complex processes, some of which are unconscious: mutual attentiveness (e.g., mutual gaze, mutual interest, and focus during interaction), positivity (e.g., head nods, smiles, friendliness, and warmth) and unconscious coordination (e.g., postural mirroring, synchronized movements, balance, and harmony) (Grahe, 1999; Tickle-Degnen and Rosenthal, 1990).

In this article, we focus on one of these processes, coordination, of partial subtle mirroring, and synchronized movements of facial expressions, and head movements (Fischer and Hess, 2017; Hess and Fischer, 2014; Chartrand et al., 2005). Mirroring has been studied extensively in interactions between robots and humans, as well as in interactions between ECAs and humans. However, there are only a few studies (Duque-Domingo et al., 2020; Cavedon et al., 2015) that examine interactions between humans and ECAs that are integrated with robots, and none of them examine the effect of mirroring nonverbal behaviors in such interactions.

Our aim is to answer the research question as to whether integrating an ECA capable of mirroring its interlocutor’s facial expressions and head movements (continuously or intermittently) with a human-service robot will improve the user’s experience with the support robot that is capable of performing useful mobile manipulative tasks (e.g. at home). Our current contribution reviews latest research on our topic, and discusses our approach to modeling rapport for human-robot interaction. We discuss the integration of our speaking, expressive, and realistic ECA with a robotic platform, the Toyota Human-Support Robot (shown in Figure 1), and how we enabled the ECA to sublty track its interlocutor’s face, and to mirror their facial expressions and head movements in real time. Lastly, we describe a pilot study we conducted towards answering our research question, which shows promising results for our forthcoming larger user study.

In human-robot interaction (HRI), research on robots establishing rapport is under way, with some research groups investigating verbal (Dieter et al., 2019; Seo et al., 2018; Grigore et al., 2016), and non-verbal (Riek et al., 2010; Ritschel et al., 2019; Hasumoto et al., 2020) cues. Previous work has examined how mimicry affects human-ECA interaction and human-robot interaction, but not human interaction with an ECA running on a robot. Hasumoto et al. (Hasumoto et al., 2020) studied the effects of body movement mimicry in human-robot interaction by designing the Reactive Chameleon, a method of generating robot body movements that subtly mimics human body swaying during interactions. They found that subtle mimicry can positively impact the establishment of rapport, while noticeable mimicry can negatively impact it. However, this method was limited to mimicking movements of the torso and did not consider movement of other parts of the robot such as the head.

In the experiment by Riek et al. (Riek et al., 2010), participants interacted with a robotic chimpanzee named Virgil that exhibited three different types of behavior: full mimicry of head gestures, partial mimicry of nodding gestures only, and no mimicry accompanied by periodic blinks. Afterwards, participants filled out a survey that measured the social attraction toward and emotional credibility of conversation partners. No significant differences were found between participant ratings of the different mimic conditions. However, this might have been due to technical issues, as a few participants “said that the head movements were too erratic or jerky.” Other participants wished for the robot to make “non-speech sounds” (backchannel cues) to indicate understanding in conjunction with head gestures. This suggests that robot mimicry of nonverbal behavior by itself might not be enough to create rapport during an interaction with a human.

Niewiadomski et al. (Niewiadomski et al., 2010) studied how mimicry of smiles influenced interactions between ECAs and humans by testing three different types of ECA smiling behavior when providing backchannel cues: mimicking the smiles of a participant (MS), randomly smiling (RS), and no smiling (NS). They found that participants felt less engaged and more frustrated in condition NS than in condition MS, and “felt more at ease and more listened to” while telling a story to the ECA in condition MS than RS. These results suggest that mimicry in the smiling behavior of an ECA influences “the quality, ease, and warmth, of the user-agent interaction.”

In the first experiment of the case study by Stevens et al. (Stevens et al., 2016), participants read some sentences and then listened to an ECA speak some of those sentences incorrectly. They were then asked to say the correct version of the sentence to the ECA. Afterwards, when interacting with the experimental group, the ECA repeated what the subject said while mirroring the eyebrow raises and head nods that were observed during the subject’s reading and correcting of potentially erroneous sentences, whereas in the control group no mimicry occurred when the sentence was repeated. The results showed that more prominent cues lead to higher ratings of ECA lifelikeness in the mimic condition, which supports the use of mimicry for building rapport in human-ECA interaction.

Although there exists previous work that integrates an avatar on a robot, to the best of our knowledge no study examines how the mimicry of nonverbal behavior by an ECA running on a robot influences interactions with humans. For example, Domingo et al. (Duque-Domingo et al., 2020) projected an avatar on a robotic head and designed a gaze control system that enabled the robotic head to reorient its position based on the location of people that it interacted with. However, the study did not investigate mimicry of nonverbal behavior by the avatar or by the robot.

We propose to integrate the Toyota Human Support Robot (HSR) (Yamamoto et al., 2019) with a fully autonomous ECA. HSR is a social robot designed to assist people with disabilities and the elderly with household tasks such as cleaning or bringing objects. HSR has a wide array of sensors that provide rich data on its surrounding environment. By using the Robot Operating System (ROS), which provides services such as access of sensor data from HSR via ROS topics, we can create modules that utilize the sensor data to drive robot behavior during interaction with humans. For an ECA, we use the modular framework eEVA (Polceanu and Lisetti, 2019), which enables the creation of ECA dialogs suitable for a wide range of scenarios.

Material. All three pilot experiments were performed under the following setup: EmoPy and DLib run on an HP Spectre laptop with 16 GB RAM, four CPUs (Intel Core i7-6500U @ 2.50GHz), and an integrated graphics unit (Intel HD Graphics 520, Skylake GT2). The ECA was created using the Unity game engine and runs on HSR, which has a CPU board (Intel Core i7-4700EQ @ 2.4GHz) and an NVIDIA Jetson GPU. Communication between the laptop and HSR in close proximity (5 meters) is facilitated using a 5G WiFi network with an average bandwidth speed of 5 ms.

Participants. All three pilot experiments were conducted with three participants recruited from our lab, ages 24 to 56 years. Participants were 2 (66%) males and 1 (33%) female, with mean age of 35 years (SD = 18.2 years). One participant reported his race as Asian (33%), and two reported their race as White (66%). Their education level was Graduate Degree (100%).

Procedure (in common). All three pilot experiments were conducted with the ECA running on the HSR as shown in Fig. 1. The HSR was set up in one spot, and lighting conditions remained unchanged. Depending upon the experiment, participants were asked to take various positions while interacting with the ECA-HSR. After interacting with the ECA-robot, participants were asked to answer questions, and an informal discussion followed. We enabled and disabled modules for each corresponding behavior to control our independent variables.

Procedure (cont.) for Exp. 1. Participants were asked to stand in front of the robot (60 cm approx.), to turn their head to the sides, as well as to move their body, with the restriction that their face should not be turned more than 90 degrees from the camera. This is because DLib can detect faces from front and angled profiles, but not from a complete side profile. Each mode of interaction lasted 15 seconds. At the end of the session, we asked ”Which of the three presented versions seemed most comfortable or natural during the interaction? Why?”

Results. All participants identified the interaction with the ECA on HSR to be the most natural in the case where only the robot head was moving. They also noted that, in the case where only the ECA moved, it would feel unrealistic if they were walking around the room. The combination of the ECA and the robot head moving in synchrony put the line of sight of the robot away from the participant, which resulted in an impression that the interaction was unnatural.

Procedure (cont.) for Exp. 2. Each participant were asked to stand in the same proximity to the HSR as before (60 cm), but this time they were asked not to walk around the lab during the experiment. Each person was asked to while looking at the robot, randomly portray three expressions for 10 seconds (to give the system enough time to accomplish the emotion mirroring): happiness, anger, and neutral. Each mode of interaction lasted 20 seconds. Participants stood in front of the HSR’s head camera so that their faces would be close enough for expression detection. After the interaction was complete, participants were asked ”Which of the two presented versions did you find more engaging? Why?”

Results. All participants expressed that, although it was exaggerated and funny at times, the interaction with the ECA on HSR was more engaging when emotive facial expression mirroring was enabled. They also noted that sometimes the emotion mirrored by ECA was not the same emotion they were portraying.

Procedure (cont.) for Exp. 3. Participants were asked to walk to the side of the robot while maintaining a close proximity, and to express three different emotions on their face (happiness, anger, neutral) every five to ten seconds, while facing the robot camera. They were given no time restriction on how long they needed to interact with either setup, and asked to give a verbal cue when they wished to end the interaction. When they stopped, they were asked ”Which version of the two presented versions (posture mimicking and emotion mirroring both enabled, or neither) did you prefer? Why?”

Results. All participants preferred the condition where both posture mimicking and emotive mirroring were enabled. They explained that the two behaviors made the ECA and the robot more engaging, and helped to establish a connection with the ECA and the robot.

To our knowledge, our proposed approach is one of the earliest, if not the first, to study how mimicry of nonverbal behavior by a fully autonomous ECA integrated with a fully autonomous physical service robot influences the agents’ interactions with humans. The preliminary experiments conducted on rapport-building – specifically on posture mimicking and emotive mirroring – show promising results.

In the first experiment on posture mimicking, participants preferred the robot head moving alone without extra movement from the ECA. In the second experiment on emotive mirroring, participants felt more engaged when the ECA mirrored their facial expression, and reported that the interaction felt more natural. Finally, in the third experiment that investigated the combined behaviors of posture mimicking and emotive mirroring, participants chose to interact longer when both behaviors were enabled in comparison to when neither were enabled.

For future work, we will consider alternatives to EmoPy to improve the emotive mirroring behavior of our ECA. More importantly, we plan to conduct these experiments beyond this pilot with a large enough N and validated measures to produce statistically significant results that can be generalized.

The video demonstration of the experiments can be found at https://www.cs.miami.edu/home/visser/hsr-videos/nHRI21.mp4.