ESIHGNN: Event-State Interactions Infused Heterogeneous Graph Neural Network for Conversational Emotion Recognition

In a conversation, a series of $N$ sequential utterances are encountered, denoted as $[(u_{1},p_{u_{1}}),(u_{2},p_{u_{2}}),\ldots,(u_{N},p_{u_{N}})]$. Here, $u_{i}$ represents the utterance of the $i$-th turn, and $p_{u_{i}}$ indicates the speaker identity for $u_{i}$. Note that $p_{u_{i}}$ and $p_{u_{j}}$ may belong to the same speaker, but there must be at least two speakers participating in the conversation. The task of the CER is to predict the emotion category for each utterance. For a target utterance $u_{i}$, its prediction relies on real-time sequence pairs consisting of utterances and accompanying speaker identities: $\left\{(u_{1},p_{u_{1}}),(u_{2},p_{u_{2}}),\ldots,(u_{i},p_{u_{i}})\right\}$.

A new methodology for modeling conversation structure is presented, emphasizing the interactions between events and speakers’ emotional states. This methodology constructs a heterogeneous event-state interaction graph, denoted as $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{R},\mathcal{A})$, where $\mathcal{V}$ represents the node set consisting of event nodes $\mathcal{V}_{e}$ and state nodes $\mathcal{V}_{s}$. In this graph, an edge $e_{ij}=\left(v_{i},r,v_{j}\right)\in\mathcal{E}$ signifies a connection from a node $v_{i}\in\mathcal{V}$ to its neighboring node $v_{j}\in\mathcal{V}$ under relation type $r\in\mathcal{R}$, with its representation denoted by $a_{ij,r}\in\mathcal{A}$.

Event Node Updating:

• •

  xWant: Intra-speaker $event\texttt{-}to\texttt{-}event$ interaction: Event $e_{i}$ passes the speaker’s action guidance to the subsequent event $e_{j}$.

• •

  oWant: Inter-speaker $event\texttt{-}to\texttt{-}event$ interaction: Event $e_{i}$ can coordinate and motivate the subsequent event $e_{j}$ triggered by another speaker.

• •

  xDrive: Intra-speaker $state\texttt{-}to\texttt{-}event$ interaction: The speaker’s emotional state $s_{i}$ internally influences the trend of their subsequent event $e_{j}$.

• •

  oDrive: Inter-speaker $state\texttt{-}to\texttt{-}event$ interaction: The speaker’s emotional state $s_{i}$, resulting from contagion and empathy, externally influences the trend of the subsequent event $e_{j}$ triggered by another speaker.

State Node Updating:

• •

  xReact: Intra-speaker $event\texttt{-}to\texttt{-}state$ interaction: When event $e_{i}$ is executed by the speaker, their subsequent state $s_{j}$ is influenced by their reaction to this event $e_{i}$.

• •

  oReact: Inter-speaker $event\texttt{-}to\texttt{-}state$ interaction: The reaction of another speaker to an event $e_{i}$ influences their subsequent state $s_{j}$.

• •

  xDepend: Intra-speaker $state\texttt{-}to\texttt{-}state$ interaction: The speaker’s emotional state $s_{i}$ at a given moment has an influence on their subsequent emotional state $s_{j}$, indicating self-dependency.

• •

  oDepend: Inter-speaker $state\texttt{-}to\texttt{-}state$ interaction: The emotional states of different speakers influence and interact with each other, indicating inter-speaker dependency.

We will now describe the methodology for feature transformation using HDAGNN. In addition to establishing event-state interactions across different turns for conversational coherence (see Subsection 2.2), it is equally crucial to consider the event-state interactions within the same turn to improve conversational consistency. To achieve this, we propose establishing two implicit, bidirectional paths connecting the event and its accompanying emotional state.

Once obtained, the normalized coefficients are used to compute a linear combination of the features of $v_{i}$’s neighborhood $\mathcal{N}_{i}$, producing aggregated information (referred to as “message”):

In HDAGNN, two GRUs are used to address the node heterogeneity and capture the sequential dependencies among nodes in an event-state interaction graph. The GRU takes the past feature $h_{i}^{l-1}$ of the node $v_{i}$ as its input and the message $M_{i}^{l}$ as its hidden state.

The second path involves the emotional state of the speaker at the $i$-th turn being influenced by the context of the event, $M_{e_{i}}^{l}$:

To ensure conversational coherence and consistency, the hidden feature of the node $v_{i}$ at the $l$-th layer is the addition of $\bar{h}_{i}^{l}$ and $\tilde{h}_{i}^{l}$:

To preserve the original semantics of each node $v_{i}$, we employ a concatenation operation $\|$ to combine its hidden features from each layer $l$ in HDAGNN, resulting in the final node representation $H_{v_{i}}$. Additionally, considering node consistency within a conversation turn $i$, we take the sum of the event representation $H_{e_{i}}$ and the state representation $H_{s_{i}}$ as the emotion representation $H_{i}$ of the utterance $u_{i}$, and input it into a fully connected layer to predict the emotion label:

The results of the four datasets are summarized in Table 2. It can be observed that: (1) Our proposed ESIHGNN achieves state-of-the-art performance, except for competitive results in MELD. This is probably attributed to the complex and ambiguous contexts present in short conversations with multiple speakers in MELD, which limit ESIHGNN’s capability to effectively capture event-state interactions. (2) Among recurrence-based methods, COSMIC outperforms DialogueRNN when external knowledge is incorporated, while our method surpasses DAG-ERC among graph-based methods. These findings highlight the importance of incorporating external knowledge. (3) Our ESIHGNN surpasses graph-based methods that do not consider future utterances on all datasets, confirming the superiority of the event-state connection structure and the effectiveness of ESIHGNN. Although considering future utterances can enhance the model’s understanding of the current context [27, 17], it is not practical for real-time conversations. (4) Graph-based methods consistently outperform recurrence-based methods on most CER datasets, indicating their superior capability to model conversational context. Our method achieves better results compared to existing graph-based methods on IEMOCAP and EmoryNLP, showing its superiority in modeling conversation, particularly for longer conversations (with an average length of 50) in IEMOCAP.

In our ablation study, we analyze the contributions of different components in our ESIHGNN: edge construction, edge representation, IntraESI (Intra-turn Event-State Interactions), and window size $\omega$. The edge construction module distinguishes our work from previous approaches like DAG-ERC [15] and SKAIG [17], as it allows the analysis of relations at a coarser scale. For example, $\texttt{-}\{event\texttt{-}to\texttt{-}event\}$ denotes the removal of the xWant and oWant relations. To evaluate the effect of different edge representations, we use “ ​trainable ​”, which replaces all edge representations with trainable embeddings, and “ ​0/1”, which limits the relation types to the set $\mathcal{R}=\{0,1\}$. Table 3 presents the ablation results. It can be seen that: (1) Removing any of the coarse-grained relations in ESIHGNN leads to a decrease in overall performance, particularly in participants’ reactions to events (i.e., $event\texttt{-}to\texttt{-}state$). This suggests that emotion recognition is influenced by both previous events and speakers’ emotional states. (2) Removing knowledge-based edges results in a more significant performance decline compared to removing trainable embedding-based edges. Additionally, both “ ​trainable ​” and “ ​0/1” show poor performance without encoding external knowledge in edge representations. These findings suggest that external knowledge can enhance the information interaction between nodes. (3) Ablating the IntraESI module noticeably decreases the performance of ESIHGNN, suggesting that there is an interplay between the speaker’s events and emotional states. (4) Increasing $\omega$ may not significantly affect the performance of ESIHGNN, indicating that the event-state interactions are localized.