Analysis and Detection of Differences in Spoken User Behaviors between Autonomous and Wizard-of-Oz Systems

ERICA [3, 4] is an autonomous android equipped with the capability to engage proficiently in diverse social contexts. ERICA has a human-like appearance complemented by a synthesized voice, trained on the vocal characteristics of a Japanese voice actress. ERICA generates a range of linguistic phenomena, including backchannels, fillers, and laughter. Furthermore, ERICA produces a variety of visual cues, encompassing facial expressions, synchronized lip movements during speech, blinking, and nodding gestures. ERICA showcases versatility by assuming distinctive roles, including attentive listening [5], conducting practice job interviews [6, 7], and serving as a laboratory guide [8]. The primary objective of ERICA is to replicate human-like communication abilities across a broad range of scenarios.

Attentive listening involves the system actively engaging in speech perception, attentively listening to the subject’s spoken communication. Attentive listening with conversational robots offers notable benefits, particularly in offering companionship to elderly individuals [5] and patients in psychiatric daycare settings [9]. These systems demonstrate attentiveness by generating backchannels, asking follow-up questions, and offering empathetic responses.

During the sessions, subjects engaged in approximately 5 to 8-minute conversations with the robot ERICA, which was operated either autonomously or remotely piloted by an operator. Conversation topics varied, with participants discussing subjects such as food, travel, or the challenges posed by the COVID-19 pandemic. The attentive listening system used in this work was developed by [5].

In the job interview scenario, the robot ERICA assumes the role of the interviewer, conducting a practice interview with the interviewee—the user. Job interviews are widely recognized as challenging and can evoke anxiety in candidates. Interacting with an autonomous system offers prospective employees and students a valuable opportunity to practice and ready themselves for interviews, thereby mitigating performance-related anxieties. Each session spanned approximately 9 minutes and was operated autonomously or remotely piloted by an operator. The job interview system in this study was based on the work of [6], integrating follow-up questions that required further elaboration from the interviewee.

This work builds upon the research conducted by [10], which similarly explored the analysis and evaluation of user behaviors in SDSs. The current study evaluated several spoken user behaviors, including fillers, backchannels, disfluencies, and laughter. While ERICA is capable of generating some of these phenomena as well, this study focused solely on the subjects’ spoken behaviors.

Fillers play a crucial role in turn-taking and floor maintenance [11], and can serve as sociolinguistic identifiers [12]. Examples of fillers include “ano” and “eto” in Japanese, and “uh” and “um” in English. In this work fillers are generated using the approach proposed by [13].

Backchannels convey attentiveness, interest, understanding, acknowledgement, agreement, and importantly, do not take the conversational floor [14]. Active listening can be demonstrated through backchannels, elevating the listener to co-narrator status [15], and adaptive backchanneling has been observed to encourage less talkative participants to engage more actively in the conversation [16]. Examples of backchannels include “hai” and “un” in Japanese, and “mhmm” and “uh-huh” in English. Backchannel frequency significantly differs between these two languages, with Japanese incorporating backchannels much more frequently than American English [17]. Previous research has explored the generation of verbal backchannels in both dyadic conversations [18] and group settings [19]. This study focused on verbal backchannels, as opposed to nonverbal backchannels like nodding.

Disfluencies include a range of phenomena, such as lengthening, truncation, repair, or word fragmentation. These phenomena can have both positive and negative effects for listeners. For example, [20] found that material preceded by a disfluency is more likely to be remembered, and [21] found that disfluencies can provide additional processing time to listeners through temporal delay. However, disfluencies can also carry negative implications, potentially leading listeners to perceive the speaker as less prepared or less knowledgeable.

Laughter functions as a tool in fostering social relationships in both human-human and human-robot contexts. Laughter is pivotal for SDSs, contributing to emotional and affective engagement, natural language understanding, and pragmatic reasoning [22]. Additionally, mutual laughter has been linked to positive outcomes during job interviews, including increased likelihood of job offers for interviewees [23], and the establishment of rapport [24]. ERICA generates laughter using the approach outlined by [25].

We used inter-pausal units (IPU) as the transcription unit, with speech segments divided into distinct IPUs whenever a pause of 200 ms or longer was detected. Furthermore, the transcription data was manually annotated to include additional linguistic information, including backchannels, fillers, disfluencies, and laughter. Text tokenization into characters was performed using MeCab¹¹1https://taku910.github.io/mecab/. Length and speaking rate were measured using characters rather than words, as it offers a more stable metric. Length was quantified as the number of tokenized characters within each IPU, and speaking rate was calculated by dividing the length of the IPU by its duration, measured in seconds.

This study investigated attentive listening and job interview scenarios for both WoZ and autonomous conditions. During the WoZ condition, operators spoke into a microphone, and the audio was played through ERICA’s speaker. When users interacted with ERICA, a wide range of audio and video data was captured. All interactions were conducted in Japanese. Figure 1 show an example of the experimental setup.

Our analysis revealed that subjects exhibit longer IPU lengths and speak at a faster rate (with greater variability) in the WoZ condition compared to the autonomous condition. This observed increase in the attentive listening scenario suggests that the WoZ condition effectively encourages subjects to speak more frequently and at a faster pace.

For fillers, we observed an increase in usage for the WoZ condition in the attentive listening scenario. This phenomenon could stem from the self-driven nature of the attentive listening task. We also noted an increase in both the mean and standard deviation of fillers per second for the WoZ condition in the attentive listening scenario. This suggests that the WoZ operator is adept at displaying natural behaviors and eliciting a higher usage of fillers per second from the subjects.

We found an increase in backchannel presence and a higher rate of backchannels per second for the WoZ condition for attentive listening. This indicates that subjects exhibit more active listening behaviors, even when engaging in predominantly one-sided conversations with the WoZ operator, compared to interactions with the robot.

We noted a reduction in disfluency usage and the frequency of disfluencies during the attentive listening scenario. This decrease could be attributed to the WoZ operator being less distracting and eliciting fewer disfluency errors compared to the robot.

We observed a significant increase in the subject’s usage of laughter, accompanied by an increase in both the mean and standard deviation for the frequency of laughter, in the WoZ condition during the attentive listening scenario. This suggests that the WoZ operators are more capable of creating an environment for eliciting laughter in these situations.

Our analysis indicated that subjects use shorter IPU lengths (with less variability) in the WoZ condition compared to the autonomous condition. An increase in speaking rate was observed in this scenario, which could potentially be attributed to nervousness caused by the job interview scenario.

We noted a significant decrease in filler usage for the WoZ condition in the job interview scenario. Given the demanding nature of job interviews, it is plausible that subjects would strive to minimize their use of fillers, with the WoZ operator better simulating real-world job interview conditions and thereby prompting subjects to employ fewer fillers. We also found a minor decrease in the average fillers per second, coupled with a substantial increase in standard deviation for the WoZ condition. The large standard deviation suggests considerable variability among subjects in their utilization of fillers when interacting with the WoZ operator, potentially reflecting varying degrees of nervousness among individuals.

We found an increase in the occurrence of backchannels and an elevated rate of backchannels per second for the WoZ condition in the job interview scenario. This substantial increase in backchannel usage suggests that participants may exhibit greater interest when conversing with the WoZ operator compared to the autonomous system.

We observed an overall decrease in disfluency usage during the job interview scenario for the WoZ condition. This reduction could be attributed to the WoZ operator being less distracting, which may have led the subject to produce fewer disfluencies compared to when interacting with the autonomous system. However, it is noteworthy that disfluencies and disfluencies per second were the only features in the job interview scenario that did not exhibit significant differences between the conditions.

We found a large increase in the subject’s usage of laughter, and in the mean and standard deviation for the frequency of laughter, in the WoZ condition for the job interview scenario. This suggests that the WoZ operator can cultivate a more humorous atmosphere even within the context of a job interview. Such elements are significant in an interview context, as laughter and humor can serve as effective icebreakers, alleviate tension, and build rapport.

We also conducted predictive modeling on both the attentive listening and job interview data. The goal was to predict whether the subject was speaking to an operator (WoZ) or the autonomous system via information gathered from the subject for each IPU. The predictive features were the same for all models and included the following: number of backchannels, backchannels per second, number of laughs, laughs per second, number of fillers, fillers per second, number of disfluencies, disfluencies per second, IPU length, speaking rate, and mean and standard deviation for both f0 and power. All measurements were calculated for each IPU. Separate training-test splits were generated for both the attentive listening and job interview datasets. Random sampling was used to generate an 80/20 training-test split, with a preset seed value to ensure reproducibility. To prevent speaker identification from influencing the results, the subject’s IPUs were allocated exclusively to either the training split or the test split. The default hyperparameters were used for all modeling, and no further tuning was conducted.

Evaluation metrics for all models can be found in Table 2 for the attentive listening and job interview scenarios. We evaluated several models: a baseline model, a logistic regression (lg) model, a support vector machine (svm) model with a Gaussian radial basis function kernel, and a random forest (rf) model. Our metrics included accuracy (acc), precision (prec), recall, and F1 score. The baseline model predicted the majority class for all instances in the test data (i.e., a recall score of 1). For both the attentive listening and the job interview scenarios, all models performed better than the baseline in terms of accuracy and precision. For F1 score, the SVM model outperformed the other models for the attentive listening scenario, while the random forest model outperformed the other models in the job interview scenario.

To investigate the impact of each predictor, we conducted permutation-based variable importance analysis on the random forest models for both the attentive listening and job interview datasets. This analysis quantified the significance of each feature by computing the change in model performance (i.e., accuracy) by randomly permuting the feature in question while keeping the other features constant. Lower importance scores indicate a smaller effect on model performance, whereas higher scores denote a larger impact. The scale of importance scores can vary depending on factors like the dataset, model, and evaluation metric employed. Thus, it is important to interpret these scores relative to each other rather than focusing solely on their absolute values. We conducted this analysis separately for each dataset, yielding permutation feature importance scores for each predictor variable. Table 3 shows the rankings of feature importance.

In both the attentive listening and job interview datasets, acoustic features emerged as the primary drivers for the model’s predictive performance. We also evaluated an acoustic-only model and linguistic-only model. The acoustic-only model performed slightly worse than the combined acoustic and linguistic features model, while the linguistic-only model performed significantly worse. These findings collectively suggest that, although linguistic features are beneficial, they are insufficient on their own for accurately classifying whether the subject is interacting with an operator or an autonomous system at the IPU level.