Graph Based Network with Contextualized Representations of Turns in Dialogue

Relation extraction has been studied extensively over the past few years and many approaches have achieved remarkable success. Most previous approaches focused on sentence-level RE (Zeng et al., 2014; Wang et al., 2016; Zhang et al., 2017; Zhu et al., 2019), but recently cross-sentence RE has been studied more because a large number of relational facts are expressed in multiple sentences in practice.

Recent work begins to explore cross-sentence relation extraction on documents that are formal genres, such as professionally written and edited news reports or well-edited websites. In document-level RE, various approaches including transformer-based methods (Tang et al., 2020; Ye et al., 2020; Wang et al., 2019) and graph-based methods (Christopoulou et al., 2019; Nan et al., 2020; Zeng et al., 2020) have been proposed. Among these, graph-based methods are widely adopted in document-level RE due to their effectiveness and strength in representing complicated syntactic and semantic relations among structured language data. Unlike previous work, we focused on extracting relations from dialogues, which are texts with high pronoun frequencies and low information density.

(Yu et al., 2020; Xue et al., 2021) were among the early works on dialogue-based RE. Yu et al. (2020) introduced several dialogue-based RE approaches with the DialogRE dataset. Among the various approaches, BERT_s, a model that uses BERT (Devlin et al., 2019), shows good performance. BERT_s is a model that slightly modified the original input sequence of BERT in consideration of speaker information. However, it has a limitation in that it cannot predict asymmetric inverse relations well. Our model basically follows the input sequence of BERT_s, but we designed it to overcome this limitation to some extent. More detailed explanation is in Sec 4.1.5. Xue et al. (2021) proposed a graph-based approach, GDPNet, that constructs a latent multi-view graph to capture various possible relationships among tokens and refines this graph to select important words for relation prediction. In this approach, the refined graph and the BERT-based sequence representations are concatenated for relation extraction. The graph of GDPNet is a multi-view directed graph aiming to model all possible relationships between tokens. Unlike GDPNet, we combine tokens into meaningful units to form nodes and connect the nodes with speaker edges, dialogue edges, and argument edges to model what each edge means. In addition, GPDNet focuses on refining this multi-view graph to capture important words from long texts for RE, but we extract the relations using the features of the nodes in the graph.

Emotion recognition in conversation has emerged as an important problem in recent years and many successful approaches have been proposed. In ERC, numerous approaches including recurrence-based methods (Majumder et al., 2019; Ghosal et al., 2020) and graph-based methods (Ghosal et al., 2019; Ishiwatari et al., 2020) have been proposed. For instance, DialogueRNN (Majumder et al., 2019) uses an attention mechanism to grasp the relevant utterance from the whole conversation and models the party state, global state, and emotional dynamics with several RNNs. COSMIC (Ghosal et al., 2020) adopts a network structure, which is similar to DialogueRNN but adds external common sense knowledge to improve performance. DialogueGCN (Ghosal et al., 2019) treats each dialogue as a graph where each node represents utterance and is connected to the surrounding utterances. RGAT (Ishiwatari et al., 2020) is based on DialogueGCN. It adds relational positional encodings that can capture speaker dependency, along with sequential information. Many studies with remarkable success have been proposed, but none can be used in ERC as well as other dialogue-based tasks like our approaches.

We follow BERT_s (Yu et al., 2020) as the input sequence of the encoding module. Given a dialogue $d=s_{1}:t_{1},s_{2}:t_{2},...,s_{M}:t_{M}$ and its associated argument pair $(a_{1},a_{2})$, where $s_{i}$ and $t_{i}$ denote the speaker ID and text of the $i^{th}$ turn, respectively, and $M$ is the total number of turns, BERT_s constructs $\hat{d}=\hat{s}_{1}:t_{1},\hat{s}_{2}:t_{2},...,\hat{s}_{M}:t_{M}$, where $\hat{s}_{i}$ is:

$$\hat{s}_{i}=\begin{cases}[S_{1}]&\text{if }s_{i}=a_{1}\\ [S_{2}]&\text{if }s_{i}=a_{2}\\ \hfil s_{i}&\text{otherwise}\end{cases}$$

(1)

where $[S_{1}]$ and $[S_{2}]$ are special tokens. In addition, it defines $\hat{a}_{k}(k\in\{1,2\})$ to be $[S_{k}]$ if $\exists i(s_{i}=a_{k})$, and $a_{k}$ otherwise. Then, we concatenate $\hat{d}$ and $(\hat{a}_{1},\hat{a}_{2})$ with a classification token [CLS] and a separator token [SEP] in BERT(Devlin et al., 2019) as the input sequence [CLS]$\hat{d}$[SEP]$\hat{a}_{1}$[SEP]$\hat{a}_{2}$[SEP].

To model the speaker change information, following SA-BERT(Gu et al., 2020), we add additional speaker embeddings to the token representations. $E_{s}(\hat{s}_{i})$ is added to each token representation of $\hat{s}_{i}:t_{i}$, $E_{s}(\hat{a}_{k})(k\in\{1,2\})$ is added to each token representation of $\hat{a}_{k}$ if $\hat{a}_{k}=[S_{k}]$, and $E_{s}(\sharp)$ is added to all token representations without speaker embedding added, where $E_{s}(\cdot)$ denotes the speaker embedding layer. $E_{s}(\sharp)$ is an embedding output for token representations without speaker information. A visual architecture of our input representation is illustrated in Appendix.

Then, token representations containing speaker change information are fed into an encoder to extract the speaker-sensitive token representations. The encoder can be BERT or BERT variants (Liu et al., 2019; Conneau and Lample, 2019; Lan et al., 2020).

To obtain the turn context-sensitive representation for each turn, we apply Masked Multi-Head Self-Attention (Vaswani et al., 2017) to the output of the encoder using the surrounding turn mask. The range of this surrounding turn is called the window, and the number of turns from the front and rear are viewed as the surrounding turn which is called the surround turn window size. The surround turn window size $c$ is a hyper-parameter.

Let $X=[x_{1},x_{2},x_{3},...,x_{N}]$ be an output of the encoding module, where $x_{j}$ is the $j^{th}$ token representation in the output and N is the number of tokens. For token representations corresponding to $\hat{s}_{i}:t_{i}$ range from $x_{b_{i}}$ to $x_{e_{i}}$, $D_{i}=[x_{b_{i}},x_{b_{i}+1},...,x_{e_{i}}]$ denotes representations of the $i^{th}$ turn, $D=[x_{b_{1}},x_{b_{1}+1},...,x_{e_{M}}]$ denotes representations of a dialogue $\hat{d}$, and $F(x_{m})$ denotes the turn number in which $x_{m}$ is included (e.g., $F(x_{m})=2$ if $x_{m}\in D_{2}$). We implement the surrounding turn mask as follows:

$$M^{sur}_{mn}=\begin{cases}\hfil 1&\text{if }x_{m}\notin D,m=n\\ \hfil-\infty&\text{if }x_{m}\notin D,m\neq n\\ \hfil 1&\text{if }x_{m}\in D,x_{n}\in R(x_{m})\\ \hfil-\infty&\text{otherwise}\end{cases}$$

(2)

where $R(x_{m})$ denotes $\bigcup\limits_{z=F(x_{m})-c}^{F(x_{m})+c}D_{z}$. A visual architecture of an example regarding the surrounding turn mask is illustrated in Appendix.

Then, we reinforce the representation of each turn from representations of surrounding turns.

To model the dialogue-level information, interactions between turns and arguments, and interactions between turns, a heterogeneous dialogue graph is constructed.

We form four distinct types of nodes in the graph: dialogue node, turn node, subject node, and object node. The dialogue node is a node with the purpose of containing overall dialogue information. Turn nodes represent information about each turn in the dialogue and are created as many as the total number of turns in the dialogue. The subject node and object node represent the information of each argument. In our work, the initial representation of the dialogue node uses a feature corresponding to [CLS] in the output of the turn attention module. The initial representation of the $i^{th}$ turn node, subject node, and object node use the average of the token representations corresponding to $\hat{s}_{i}:t_{i}$, $\hat{a}_{1}$, and $\hat{a}_{2}$ in the output of the turn attention module, respectively.

There are three different types of edge:

• •

  dialogue edge: All turn nodes are connected to the dialogue node with the dialogue edge so that the dialogue node learns while being aware of turn-level information.

• •

  argument edge: To model the interaction between turns and arguments, the $i^{th}$ turn node and argument nodes (i.e., subject node and object node) are connected with the argument edge if the argument is mentioned in $\hat{s}_{i}:t_{i}$.

• •

  speaker edge: To model the interaction among different turns of the same speaker, turn nodes uttered by the same speaker are fully connected with speaker edges.

Next, we apply a Graph Convolutional Network (GCN) (Kipf and Welling, 2017) to aggregate each node feature from the features of the neighbors. At this time, in order to inject sequential information to the turn nodes, GCN is applied after the turn nodes pass through the bidirectional LSTM (Schuster and Paliwal, 1997) layers. Given node $u$ at the $l^{th}$ GCN layer, $h^{(l)}_{u}$ and $\hat{h}^{(l)}_{u}$ denote the representation of the node before injecting sequential information and the representation of the node after injecting sequential information, respectively. $\hat{h}^{(l)}_{u}$ can be defined as:

$$\dot{h}^{(l)}_{T_{i}}=[\overrightarrow{LSTM^{(l)}}(h^{(l)}_{T_{i}});\overleftarrow{LSTM^{(l)}}(h^{(l)}_{T_{i}})]$$

(3)

$$\hat{h}^{(l)}_{u}=\begin{cases}W^{(l)}_{\alpha}\dot{h}^{(l)}_{T_{i}}+b^{(l)}_{\alpha}&\text{if type of }u\text{ is turn node}\\ \\ \hfil h^{(l)}_{u}&\text{otherwise}\end{cases}$$

(4)

where $T_{i}$ represents an $i^{th}$ turn node and $\dot{h}^{(l)}_{T_{i}}$ represents turn node feature injected sequential information in the dialogue by concatenating the hidden states of two directions. $W^{(l)}_{\alpha}\in\mathbb{R}^{d\times 2d}$, $b^{(l)}_{\alpha}\in\mathbb{R}^{d}$, and $d$ is the dimension. Then, the graph convolution operation can be defined as:

$$h^{(l+1)}_{u}=ReLU\left(\sum_{k\in\kappa}\sum_{v\in N_{k}(u)}W^{(l)}_{k}\hat{h}^{(l)}_{v}+b^{(l)}_{k}\right)$$

(5)

where $\kappa$ are different types of edges, $N_{k}(u)$ denotes neighbors for node $u$ connected in the $k^{th}$ type edge, $W^{(l)}_{k}\in\mathbb{R}^{d\times d}$, and $b^{(l)}_{k}\in\mathbb{R}^{d}$.

We concatenate the dialogue node, subject node, and object node to classify the relation between arguments. Furthermore, to cover features of all different abstract levels from each layer of the GCN, we concatenate the hidden states of each GCN layer as follows:

$$C=[h^{(0)}_{d};h^{(0)}_{s};h^{(0)}_{o};...;h^{(G)}_{d};h^{(G)}_{s};h^{(G)}_{o};]$$

(6)

where $G$ is the number of GCN layers and $d$, $s$, and $o$ denote the dialogue node, subject node, and object node, respectively. For each relation type $r$, we introduce a vector $W_{r}\in\mathbb{R}^{3(G+1)d}$ and obtain the probability $P_{r}$ of the existence of $r$ between arguments by $P_{r}=sigmoid(CW^{T}_{r})$. We use cross-entropy loss as the classification loss to train our model in an end-to-end way.

We conduct ablation studies to evaluate the effectiveness of different modules and mechanisms in TUCORE-GCN. The results are shown in Table 6.

First, we removed the speaker embedding in the encoder module. To be specific, the encoder and input format of TUCORE-GCN_RoBERTa are the same as RoBERTa_s. Without speaker embedding, the performance of TUCORE-GCN_RoBERTa drops by 0.4 $F1$ score and 0.1 $F1_{c}$ score on the DialogRE test set and 0.72 and 1.65 $F1$ scores on the MELD and EmoryNLP test set, respectively. This drop shows that when encoding a dialogue, a better representation can be obtained through speaker change information.

Next, we removed the turn attention module. To be specific, the output of the encoding module is delivered to the dialogue graph with sequential nodes module. Without turn attention, the performance of TUCORE-GCN_RoBERTa sharply drops by 1.1 $F1$ score and 0.6 $F1_{c}$ score on DialogRE test set and 0.77 and 2.17 $F1$ scores on the MELD and EmoryNLP test set, respectively. This drop shows that the turn attention module helps capture the representation of the turns and, therefore, improves dialogue-based RE and ERC performance.

Finally, we removed the turn-level BiLSTM for turn nodes in the dialogue graph with sequential nodes module. To be specific, in the module, we apply GCN without injecting sequential information of the turn nodes. Without turn-level BiLSTM, the performance of TUCORE-GCN_RoBERTa drops by 0.6 $F1$ score and 0.2 $F1_{c}$ score on DialogRE test set and 0.34 and 0.89 $F1$ scores on MELD and EmoryNLP test set, respectively. This means that reflecting the characteristics of the sequential nodes when learning the graph helps to learn the features of each node and, therefore, improves dialogue-based RE and ERC performance.