Directed Acyclic Graph Network for Conversational Emotion Recognition

Recently, several ERC datasets with textual data have been released (Busso et al., 2008; Schuller et al., 2012; Zahiri and Choi, 2017; Li et al., 2017; Chen et al., 2018; Poria et al., 2019b), arousing the widespread interest of NLP researchers. In the following paragraphs, we divide the related works into two categories according to the methods they use to model the conversation context.

Directed acyclic graph is a special type of graph structure that can be seen in multiple areas, for example, the parsing results of source code Allamanis et al. (2018) and logical formulas (Crouse et al., 2019). A number of neural networks that employ DAG architecture have been proposed, such as Tree-LSTM (Tai et al., 2015), DAG-RNN(Shuai et al., 2016), D-VAE (Zhang et al., 2019b), and DAGNN (Thost and Chen, 2021). DAGNN is different from the previous DAG models in the model structure. Specifically, DAGNN allows multiple layers to be stacked, while the others have only one single layer. Besides, instead of merely carrying out naive sum or element-wise product on the predecessors’ representations, DAGNN conducts information aggregation using graph attention.

In ERC, a conversation is defined as a sequence of utterances $\{u_{1},u_{2},...,u_{N}\}$, where $N$ is the number of utterances. Each utterance $u_{i}$ consists of $n_{i}$ tokens, namely $u_{i}=\{w_{i1},w_{i2},...,w_{in_{i}}\}$. A discrete value $y_{i}\in\mathcal{S}$ is used to denote the emotion label of $u_{i}$, where $\mathcal{S}$ is the set of emotion labels. The speaker identity is denoted by a function $p(\cdot)$. For example, $p(u_{i})\in\mathcal{P}$ denotes the speaker of $u_{i}$ and $\mathcal{P}$ is the collection of all speaker roles in an ERC dataset. The objective of this task is to predict the emotion label $y_{t}$ for a given query utterance $u_{t}$ based on dialog context $\{u_{1},u_{2},...,u_{N}\}$ and the corresponding speaker identity.

We design a directed acyclic graph (DAG) to model the information propagation in a conversation. A DAG is denoted by $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{R})$. In this paper, the nodes in the DAG are the utterances in the conversation, i.e., $\mathcal{V}=\{u_{1},u_{2},...,u_{N}\}$, and the edge $(i,j,r_{ij})\in\mathcal{E}$ represents the information propagated from $u_{i}$ to $u_{j}$, where $r_{ij}\in\mathcal{R}$ is the relation type of the edge. The set of relation types of edges, $\mathcal{R}=\{0,1\}$, contains two types of relation: $1$ for that the two connected utterances are spoken by the same speaker, and $0$ for otherwise.

We impose three constraints to decide when an utterance would propagate information to another, i.e., when two utterances are connected in the DAG:

The first constraint ensures the conversation to be a DAG, and the second and third constraints indicate that $u_{\tau}$ is the cut-off point of remote and local information. We regard $u_{\tau}$ as the $\omega$-th latest utterance spoken by $p(u_{i})$ before $u_{i}$, where $\omega$ is a hyper-parameter. Then for each utterance $u_{l}$ in between $u_{\tau}$ and $u_{i}$, we make a directed edge from $u_{l}$ to $u_{i}$. We show the above process of building a DAG in Algorithm 1.

An example of the DAG is shown in Figure 2. In general, our DAG has two main advancements compared to the graph structures developed in previous works Ghosal et al. (2019); Ishiwatari et al. (2020): First, our DAG doesn’t have edges from future utterances to previous utterances, which we argue is more reasonable and realistic, as the emotion of a query utterance should not be influenced by the future utterances in practice. Second, our DAG seeks a more meaningful $u_{\tau}$ for each utterance, rather than simply connecting each utterance with a fixed number of surrounding utterances.

In this section, we introduce the proposed Directed Acyclic Graph Neural Network for ERC (DAG-ERC). The framework is shown in Figure 3.

We conduct hyper-parameter search for our proposed DAG-ERC on each dataset by hold-out validation with a validation set. The hyper-parameters to search include learning rate, batch size, dropout rate, and the number of DAG-ERC layers. For the $\omega$ that is described in 3.2, we let $\omega=1$ for the overall performance comparison by default, but we report the results with $\omega$ varying from 1 to 3 in 5.2. For other hyper-parameters, the sizes of all hidden vectors are equal to 300, and the feature size for the RoBERTa extractor is 1024. Each training and testing process is run on a single RTX 2080 Ti GPU. Each training process contains 60 epochs and it costs at most 50 seconds per epoch. The reported results of our implemented models are all based on the average score of 5 random runs on the test set.

We evaluate DAG-ERC on four ERC datasets. The statistics of them are shown in Table 1.

We utilize only the textual modality of the above datasets for the experiments. For evaluation metrics, we follow Ishiwatari et al. (2020) and Shen et al. (2020) and choose micro-averaged F1 excluding the majority class (neutral) for DailyDialog and weighted-average F1 for the other datasets.

We compared our model with the following baselines in our experiments:

The overall results of all the compared methods on the four datasets are reported in Table 2. We can note from the results that our proposed DAG-ERC achieves competitive performances across the four datasets and reaches a new state of the art on the IEMOCAP, DailyDialog and EmoryNLP datasets.

As shown in the table, when the feature extracting method is the same, graph-based models generally outperform recurrence-based models on IEMOCAP, DailyDialog, and EmoryNLP. This phenomenon indicates that recurrence-based models cannot encode the context as effectively as graph-based models, especially for the more important local context. What’s more, we see a significant improvement of DAG-ERC over the graph-based models on IEMOCAP, which demonstrates DAG-ERC’s superior ability to capture remote information given that the dialogs in IEMOCAP are much longer (almost 70 utterances per dialog).

On MELD, however, we observe that neither graph-based models nor our DAG-ERC outperforms the recurrence-based models. After going through the data, we find that due to the data collection method (collected from TV shows), sometimes two consecutive utterances in MELD are not coherent. Under this circumstance, graph-based models’ advantage in encoding context is not that important.

Besides, the graph-based models see considerable improvements when implemented with the powerful feature extractor RoBERTa. In spite of this, our DAG-ERC consistently outperforms these improved graph-based models and DAGNN, confirming the superiority of the DAG structure and the effectiveness of the improvements we make to build DAG-ERC upon DAGNN.

In this section, we investigate how the structure of DAG would affect our DAG-ERC’s performance by applying different DAG structures to DAG-ERC. In addition to our proposed structure, we further define three kinds of DAG structure: (1) sequence, in which utterances are connected one by one; (2) DAG with single local information, in which each utterance only receives local information from its nearest neighbor, and the remote information remains the same as our DAG; (3) common DAG, in which each utterance is connected with $\kappa$ previous utterances. Note that if there are only two speakers taking turns to speak in a dialog, then our DAG is equivalent to common DAG with $\kappa=2\omega$, making the comparison less meaningful. Therefore, we conduct the experiment on EmoryNLP, where there are usually multiple speakers in one dialog, and the speakers speak in arbitrary order. The test performances are reported in Table 3, together with the average number of each utterance’s predecessors.

Several instructive observations can be made from the experimental results. Firstly, the performance of DAG-ERC drops significantly when equipped with the sequence structure. Secondly, our proposed DAG structure has the highest performance among the DAG structures. Considering our DAG with $\omega=2$ and common DAG with $\kappa=6$, with very close numbers of predecessors, our DAG still outperforms the common DAG by a certain margin. This indicates that the constraints based on speaker identity and positional relation are effective inductive biases, and the structure of our DAG is more suitable for the ERC task than rigidly connecting each utterance with a fixed number of predecessors. Finally, we find that increasing the value of $\omega$ may not contribute to the performance of our DAG, and $\omega=1$ tends to be enough.

To study the impact of the modules in DAG-ERC, we evaluate DAG-ERC by removing relation-aware feature transformation, the nodal information unit, and the contextual information unit individually. The results are shown in Table 4.

As shown in the table, removing the relation-aware feature transformation causes a sharp performance drop on IEMOCAP and DailyDialog, while a slight drop on MELD and EmoryNLP. Note that there are only two speakers per dialog in IEMOCAP and DailyDialog, and there are usually more than two speakers in dialogs of MELD and EmoryNLP. Therefore, we can infer that the relation of whether two utterances have the same speaker is sufficient for two-speaker dialogs, while falls short in the multi-speaker setting.

Moreover, we find that on each dataset, the performance drop caused by ablating nodal information unit is similar to contextual information unit, and all these drops are not that critical. This implies that either the nodal information unit or contextual information unit is effective for the ERC task, while combining the two of them can yield further performance improvement.

According to the model structure introduced in Section 3.3.2, the only way for GNNs to receive information from a remote utterance is to stack many GNN layers. However, it is well known that stacking too many GNN layers might cause performance degradation due to over-smoothing (Kipf and Welling, 2016). We investigate whether the same phenomenon would happen when stacking many DAG-ERC layers. We conduct an experiment on IEMOCAP and plot the test result by different numbers of layers in Figure 4, with RGAT-RoBERTa and DAGNN as baselines. As illustrated in the figure, RGAT suffers a significant performance degradation after the number of layers exceeds 6. While for DAGNN and DAG-ERC, with the number of layers changes, both of their performances fluctuate in a relatively narrow range, indicating that over-smoothing tends not to happen in the directed acyclic graph networks.

After going through the prediction results on the four datasets, we find that our DAG-ERC fails to distinguish between similar emotions very well, such as frustrated vs anger, happiness vs excited, scared vs mad, and joyful vs peaceful. This kind of mistake is also reported by Ghosal et al. (2019). Besides, we find that DAG-ERC tends to misclassify samples of other emotions to neutral on MELD, DailyDialog and EmoryNLP due to the majority proportion of neutral samples in these datasets.

We also look closely into the emotional shift issue, which means the emotions of two consecutive utterances from the same speaker are different. Existing ERC models generally work poorly in emotional shift. As shown in Table 5, our DAG-ERC also fails to perform better on the samples with emotional shift than that without it, though the performance is still better than previous models. For example, the accuracy of DAG-ERC in the case of emotional shift is 57.98% on the IEMOCAP dataset, which is higher than 52.5% achieved by DialogueRNN (Majumder et al., 2019) and 55% achieved by DialogXL (Shen et al., 2020).