I Know Your Feelings Before You Do: Predicting Future Affective Reactions in Human-Computer Dialogue

In the speech scenario, as shown in Fig. 1(a) and Algorithm 1, there are three major components in the proposed architecture: emotion prediction, emotion recognition, and self-correction. The emotion prediction model works as a future prediction function by taking the emotion and DA of the system’s current turn as input and outputting an emotion as the prediction of the user’s emotion in the next turn. When the prediction probability is high (i.e., the system is confident about its prediction), the system will take it as the user’s future emotion and use it to help plan its next turn before the user speaks. Otherwise, the emotion recognition model starts to work by detecting the user’s emotion once they finish speaking. The recognition result will be taken as ground truth to fine-tune the emotion prediction model if it has low confidence in its outputs and update the parameters of the prediction model via the self-correction function.

The emotion prediction model can be built by drawing on prior knowledge of emotion-emotion mapping (Li et al., 2019) and the DA-emotion causal relationship (Cao et al., 2021). We conducted a preliminary analysis demonstrating that in human-robot dialogue, the users indeed mimic the robot’s emotion, but the correlations show different patterns in the arousal and valence dimensions (described in Sec. 4.2). Besides, there is limited work on the causal relationship between DAs and emotions. Thus, we aim to expand this research to formulate the emotion prediction problem as a logistic regression model:

where the $EM_{prd}$ is the predicted user emotion in the next turn, and the $EM_{cur}$ and $DA_{cur}$ are the system’s emotion and DA in the current turn, respectively. The regression model can be pre-trained on suitable corpora before being incorporated in the proposed architecture.

Compared to emotion prediction, emotion recognition has been well studied, but the majority of these works did not take into consideration the real-life environment (e.g., noise), which is a long-standing problem for SDSs (Li, 2018). Hence, we propose to incorporate text as additional information to tackle this problem. The text features are extracted from transcripts generated by the ASR component in SDSs, so the emotion recognition model needs to be robust to ASR errors. In a recent study, we developed a hierarchical cross-attention fusion model using both audio and text features for ASR error-robust emotion recognition (Li et al., 2022a), which can be adopted in the proposed architecture. When using ASR transcripts, this model performed similarly to when using ground-truth transcripts.

The self-correction component follows the rule that when the confidence (i.e., probability) of the emotion prediction result is low, it starts to work by updating (i.e., fine-tuning) the prediction model using the outputs from the emotion recognition model as ground truths. Such a setting allows the emotion prediction component to dynamically adapt to the emotional expression habits of the human participants as the dialogue progresses. Like our human ability to make predictions, the longer the dialogue goes on, the more accurate the prediction becomes.

Similar to the speech scenario, there are three major components in the laughter scenario, as shown in Fig. 1(b) and Algorithm 2: laughter prediction, laughter detection, and self-correction. Based on the system’s laughter behavior in the current turn and the shared laughter relationship (Inoue et al., 2022), the laughter prediction model predicts the occurrence and type of the user’s laughter in the next turn. If the prediction probability is low, the laughter detection will work by detecting the type of laughter from the user’s response and updating the laughter prediction model via the self-correction function.

The laughter prediction model can be built by drawing on recent work that detects the occurrence and type of the user’s laughter to generate the system’s laughter (Inoue et al., 2022). We can use this finding reversely by predicting the occurrence and type of the user’s laughter based on the system’s laughter, which is easy to manipulate in SDSs. The laughter detection model takes acoustic features (e.g., MFCCs) and prosodic features (e.g., pitch and power) as input and feeds them to a stacked recurrent neural network. The recurrent neural network will be implemented using the bi-directional gated recurrent unit, whose feed-forward processing can work in real time, which is essential for SDSs. The self-correction follows the same idea as the speech scenario by updating (fine-tuning) the prediction model when its prediction probability is low.

Although there are existing emotional dialogue corpora, most of them are not suitable for the purpose of this work. For example, IEMOCAP (Busso et al., 2008) contains spontaneous dialogue sessions, yet the improvisation is limited to a set of hypothetical scenarios, which is different from open domain natural dialogue. SEMAINE (McKeown et al., 2010) collected human-agent emotional dialogues, but the human users were not permitted to ask questions. The persuasive dialogue corpus used in (Acosta and Ward, 2011) is not publicly available. Besides, none of the existing corpora contain sufficient occurrences and variations of laughter.

Therefore, we used two corpora from the JST ERATO ISHIGURO Symbiotic Human-Robot Interaction Project¹¹1https://www.jst.go.jp/erato/ishiguro/en/index.html that contain spontaneous dialogue and rich laughter. The corpus for the speech scenario consists of spontaneous dialogue between human participants and a teleoperated humanoid robot ERICA (Glas et al., 2016). During data collection, ERICA was teleoperated by a human operator in a Wizard of Oz (WoZ) manner. The participants were students ranging from 18 to 22 years old. Each dialogue session contained two phases and lasted around 15 minutes, and six sessions were conducted. In the first phase, ERICA introduced herself and talked with the participants about their lives, hobbies, and future plans. In the second phase, they talked about robots, especially about ERICA herself. During the dialogue, the robot led the dialogue, and the participant acted as a “follower”, which is the scenario we hope to apply our proposed architecture to. The emotions are annotated as Valence: -3 (extremely negative) to +3 (extremely positive), and Arousal: -3 (extremely passive) to +3 (extremely active).

The corpus for the laughter scenario was collected under almost identical conditions, except that the aim was to get the teleoperators and the participants to know each other quickly (Inoue et al., 2016). As a result, they behaved friendly by laughing frequently during such speed dating. Each dialogue lasts 10 to 15 minutes, and 82 dialogue sessions were conducted. The laughter was annotated as Social laughter, Mirthful laughter, and No laughter.

To explore the relationship between the human participant’s and the robot’s emotions in dialogue, we analyzed the human-robot dialogue sessions from the first corpus. Our preliminary analysis found similar patterns in all six sessions. Because different sessions have different durations and different numbers of utterances, we could not average the emotion labels over the six sessions. Thus, we report one session containing 123 utterance pairs as an illustrative example to discuss our findings.

The valence patterns of the robot and participant are shown in Fig. 2. The dialogue can be roughly divided into three phases. We can see that in the spontaneous dialogue phase, the human participant’s valence does mimic the robot’s to a large extent, especially when the robot changes its valence significantly (e.g., from -2 to +2 and from +1 to -3). Also, during the majority of the time in the spontaneous dialogue phase, the human valence is very close to its previous robot valence, showing a mimicry relationship. Note that, the human participant did not express extremely high valence (i.e., +3). This could be due to individual differences such as cultural background, personality, or expectation of the robot as a novel stimulus. The Pearson’s Correlation Coefficient (PCC) of the valence pairs is 0.54 in the spontaneous dialogue phase, i.e., there is a moderate positive relationship between the human and robot valences. This suggests that it is possible to implement a mapping function between the human’s valence and the valence expressed by the SDS. Note that, what we need to investigate is the correlation between emotions for a mapping pattern, not the exact values for classification, so we do not report accuracy or F1 scores. Even if the human participant’s emotion values are completely different from the robot’s, but highly correlated, e.g., [-3, -2, 1, 0, 1] and [-2, -1, 0, 1, 2], it still shows that the participant’s emotion follows the robot, which could be used as a basis for implementing our proposed anticipatory SDS.

Interestingly, during the ice-breaking and ending phases, the human’s valence hardly resembles the robot’s valence with a PCC of 0.09 in the ice-breaking phase and 0.07 in the ending phase. After examining the video recording, we found that both parties were performing the greeting and leave-taking dialogue acts that they consider to be socially appropriate, instead of mimicking their dialogue partner’s expressions. That is, emotions in the dialogue were influenced by DAs. Thus, we annotated DAs following the categorization by Stolcke et al. (2000) to understand how they impact emotion mimicry.

An annotation excerpt from the ice-breaking phase is shown in Table 1. When the robot asked a “Wh-question” and the participant responded with a “statement”, both valence and arousal display mimicry. However, when the robot expressed “signal-non-understanding” and the participant responded “reject”, the valence has an obvious drop, but the arousal barely changes. This shows that unlike valence, arousal is less influenced by DAs.

In terms of arousal, we found significant mimicry in the participant’s arousal toward the robot’s (figure omitted for brevity). The PCC of the arousal pairs is 0.78 over the whole dialogue session, including the ice-breaking and ending phases. This demonstrates that the arousal behind the future response may be relatively easy to predict based on the current utterance.

Our preliminary analysis indicates that it is feasible to predict a future emotion from the current emotion and DA for implementing the prediction component of our proposed architecture. We also found that valence is related to contextual information, such as DAs and personal factors of the interlocutor. We summarized a set of observations on the relationship between dialogue context and the valence of the robot and of the human participant in Table 2 and Table 3, respectively. We expect that these observations can contribute to the future implementation of the proposed anticipatory SDS and to the broader research community.

We analyzed a publicized dialogue demo that was built upon the second corpus with the robot’s dialogue system replacing the teleoperator²²2https://www.youtube.com/watch?v=6tMiWog4l00. The results are presented in Table 4. As shown here, the robot generated suitable laughter behaviors based on the acoustic features of the previous user laughter. When the user’s laughter had a flat pitch and moderate power, the robot responded with social laughter. When the user’s laughter had a long duration and was jittery and shimmery, the robot responded with mirthful laughter. The robot also “understood” not to laugh when the user laughed only to relieve embarrassment. Based on previous research on the contagious phenomenon of laughter (Provine, 1992), we aim to expand on the shared laughter research by adjusting the acoustic features of the laughter generated by the SDS, allowing it to predict the future laughter behaviors of the user in response to the system’s laughter.

The proposed architecture relies on accurate and robust recognition of emotions from speech, which remains an open challenge due to the variability in speech and emotion expression. The prediction component in the proposed architecture may not be applicable to all user turns, as humans can express emotions and laughter arbitrarily without considering the system’s expression. In this case, the architecture can be “downgraded” with only emotion recognition and laughter detection working and the prediction and self-correction components frozen. Further, noise in real-world applications of SDSs can reduce the reliability of both the emotion recognition and laughter detection models. Therefore, we plan to incorporate other communicative modalities (e.g., text and vision) to further improve the proposed architecture (Li et al., 2020).

In social and open-domain dialogue, an SDS with our proposed architecture can generate appropriate conversational and affective behaviors, such as a backchannel “Yeah” or laughter, in real time, instead of having delayed turn-taking that may interrupt the user’s next turn. Further, it is especially useful in scenarios where the SDS is expected to take initiatives and lead the conversation, such as in healthcare and education. For example, the SDS can adjust its generated emotions and DAs to support a user’s emotion regulation process by eliciting certain emotions in people with depression or autism (Lubis et al., 2018). In education, the SDS can express emotions during collaborative problem solving with children to increase their participation in the learning activities and resulting learning outcomes, as Zhou and Tian (2020) found that when the robots exhibited emotional expressions, participants were more likely to collaborate with them and achieve task success faster.