A Graph Isomorphism Network with Weighted Multiple Aggregators for Speech Emotion Recognition

The architecture of WMA-GIN is illustrated in Fig. 1. A Bi-GRU layer is used for extracting the global context information of the features among long-range time series, and the hidden dimension of each direction is 128. Then, the output $\mathbf{g}$ of Bi-GRU is fed into stacked WMA-GIN layers for extracting the features of higher resolution and learning the information with richer emotional characteristics. The number of WMA-GIN layers is set to 4. There are residual connections between each two adjacent WMA-GIN layers. Moreover, multi-phase attention (MPA) is employed for extracting emotional features that might be omitted from different phases of the network. Similar to self-attention [21], each of three linear projections is performed on the input features, the output of Bi-GRU, and output of the last WMA-GIN layer, respectively, such that they are transformed as $query$, $key$ and $value$. The output dimension of the linear projection is equal to the hidden size of the WMA-GIN layer.

Finally, the outputs of the Bi-GRU layer, WMA-GIN1, WMA-GIN2, WMA-GIN3, and MPA are fed into the corresponding linear layer to produce the predicted results $a$, $b$, $c$ $d$, $e$, respectively. As shown in Figure 1, all outputs of predicted results are summed up as $G$. Then, the predicted emotion label is obtained via a softmax layer.

Following the prior works [13, 14], we adopt frame-to-node transformation for our graph construction. The node feature $\mathbf{s}_{i}\in\mathbb{R}^{h}$ is obtained from the corresponding audio frame with the feature dimension of $h$. A graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}\in\left\{v_{i}\right\}_{i=1}^{n}$ is the set of $n$ nodes, and $\mathcal{E}$ is the set of all edges of each nodes. So the feature matrix of each graph is denoted as $\mathbf{S}\in\mathbb{R}^{n\times h}$. The adjacency matrix of $\mathcal{G}$ is denoted by $\mathbf{A}\in\mathbb{R}^{n\times n}$ where an element $(\mathbf{A})_{ij}$ denotes the edge weight connecting node $v_{i}$ and $v_{j}$. Here, each node has two neighboring nodes corresponding to the previous and following one frame. The relationship of node neighborhood is defined as a cycle construction where the node of the first time frame is connected with the last time frame.

The basic GIN is a GNN architecture proposed by [15] that is provably the most expressive among the class of GNNs and is as powerful as the Weisfeiler-Lehman graph isomorphism test. The GIN has achieved excellent performance for graph classification and node classification tasks. The GIN also follows a neighborhood aggregation scheme, where the representation vector of a node is computed by recursively aggregating and transforming representation vectors of its neighboring nodes [15]. The original GIN adopts a sum aggregation strategy for aggregating information representation from the neighboring nodes. The calculation of sum aggregation is as follow:

$$sum_{i}\left(X^{l}\right)=\sum_{j\in N(i)}X_{j}^{l}$$

(1)

where $X^{l}$ denotes the nodes’ features at layer $l$, and $N(i)$ the neighbourhood of node $i$.

While a single aggregator is not enough for differentiating between neighbourhood messages. For example, a node of graph $a$ receives two messages {$0.2,0.2$} from two neighbour nodes. And the received messages of another node of graph $b$ are {$0,0.4$}. In this case, the sum aggregator cannot distinguish between graph $a$ and graph $b$. For alleviating the abovementioned information confusion, we introduce two other aggregators: mean aggregator and softmax aggregator. Their calculations are as follows:

$$mean_{i}\left(X^{l}\right)=\frac{1}{d_{i}}\sum_{j\in N(i)}X_{j}^{l}$$

(2)

$$softmax_{i}\left(X^{l}\right)=\sum_{j\in N(i)}\frac{X_{j}^{l}\exp\left(X_{j}^{l}\right)}{\sum_{k\in N(i)}\exp\left(X_{k}^{l}\right)}$$

(3)

where $d_{i}=|N(i)|$ which denotes the number of neighbour nodes. The messages from different nodes can be assigned different weights through softmax aggregation. During the model training, the information will constantly flow to the part with salient emotional features. As the graph structure in our proposed SER method is relatively simple where each node gets two neighbour nodes, we do not choose to scale the results of aggregations as [19] does. Finally, the node representation in WMA-GIN updates as:

		$\displaystyle X_{i}^{(l)}=\operatorname{MLP}^{(l)}((1+\epsilon^{(l)})\cdot X_{i}^{(l-1)}$		(4)
		$\displaystyle+\alpha\sum_{j\in\mathcal{N}(i)}X_{j}^{(l-1)}+\beta\frac{1}{d_{i}}\sum_{j\in N(i)}X_{j}^{(l-1)}$
		$\displaystyle+\gamma\sum_{j\in N(i)}\frac{X_{j}^{(l-1)}\exp X_{j}^{(l-1)}}{\sum_{k\in N(i)}\exp X_{k}^{(l-1)}})$

where MLP denotes Multilayer Perceptron that is a linear layer, $X_{i}^{(l)}$ denotes feature representation of node $i$ in the $l-th$ hidden layer. And $X_{i}^{(0)}$ is initialized as $g_{i}$ which denotes the output of Bi-GRU layer. $\epsilon$ is a learnable parameter. $\alpha$, $\beta$, $\gamma$ are three hyperparameters of weights for three aggregators.

To ease the $over$-$squashing$ problem in all GNNs, we modified the second WMA-GIN layer to be a Fully-Adjacent (FA) layer. The FA layer is a WMA-GIN layer in which every pair of nodes is connected by an edge. This does not change the type of WMA-GIN layer nor add trainable parameters but only changes the adjacency relationship of the nodes in a single layer. After adding the FA layer, only the second WMA-GIN layer allows the topology-aware node-representations to interact directly and consider nodes far beyond their original neighbors [20].

A multi-stage loss training strategy is adopted in our proposed SER method. The outputs of four WMA-GIN layers and the final output are defined as five different stages in the SER network. We denote them as $b$, $c$, $d$, $e$, $G$ in Fig. 1. Generally, the stage deeper, the obtained emotional information more salient. Thus the output $G$ is assigned with the larger weights. Cross-entropy (CE) loss is used as the loss function. Note that the multi-stage loss training strategy is only used during the training phase. The calculation of the final loss function is as follows:

$$Loss=\frac{\sum_{i=1}^{I}i\cdot L_{CE}}{\sum_{i=1}^{I}i}$$

(5)

where $i/\sum_{i=1}^{I}i$ is the weight of loss at the $i-th$ stage and $I$ is the number of total stages.

We performed the experiments on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset [22], which is a popular benchmark dataset for emotion recognition. The corpus contains approximately 12 hours of data over five dyadic sessions with ten subjects. Each interaction conversation is around 5 minutes long and segmented into multiple sentences. The single categorical emotional label was assigned to every utterance, with over two of three annotators agreeing on the emotional labels. We performed the experiments on the task of four-class emotion classification for a fair comparison with other methods. The samples contain 5531 utterances (1,103 angry, 1636 happy merged with excitement, 1,084 sad, 1,708 neutral).

Follow two prior work [14, 12], we extract 78-dimensional frame-level Low Level Descriptors (LLDs) from [23] using the openSMILE toolkit [24]. We also make the experiments using 128-dimensional log-Mel spectrograms as [30] does. For each sample, we use a sliding window of length 25ms (with a stride length of 10ms) to extract the LLDs locally. We set each graph length to 120 as [13] and [14] do, that means each graph contains 120 nodes. The graph label is the same as its original utterance. An utterance may be cut as several graphs because the lengths of some utterances are much longer than 120. Padding is used to make the samples of equal length.

In the experiments, we performed 5-fold cross-validation in the speaker-independent environment, the proportion of training set, validation set, the test set was set with 8:1:1. The Adam optimizer [25] was used with the initial learning rate of 1e-4 and weight decay set to 1e-8. Moreover, the early stopping was set during training. The batch size was set to 128. We employ unweighted accuracy (UA) and weighted accuracy (WA) as evaluation metrics following the previous studies.

Fig. 2 shows the UA and WA scores of our proposed method adapting the different numbers of WMA-GIN layer along with different hidden sizes. Alon et.al [20] provide the minimal hidden size for different numbers of GIN layers to fit the training data perfectly. Therefore, we set the lower bound of hidden size to 64, 128, and 256 for four, five, and six WMA-GIN layers. The results of WA are all higher than that of UA, and both of them increase first and then decrease with the increase of hidden size. Generally, the more WMA-GIN layer we use, the worse the results are for WA or UA. The model suffers from $over$-$squashing$ that results in underfitting, which prevents the model from distinguishing between different examples. As a result, we employ the Fully-Adjacent layer in our proposed model once in the following experiments, and adopt the WMA-GIN layers of 4 with the hidden size of 256 according to the result in Fig.2.

We further explore which WMA-GIN layer should adopt the FA. Tab.1 shows three situations: the FA is applied at the last, penultimate, and antepenultimate layer. The model with the FA applied at the antepenultimate layer performs best. Those three situations of applying the FA layers can effectively ease information flow and prevent the $over$-$squashing$ problem. The model with the FA applied at the antepenultimate layer achieves a better balance between the information interaction with the emotional features extraction. So, in the following experiments, the FA was all applied at the antepenultimate layer in our proposed WMA-GIN model.

In this section, we explore different weights of three aggregators for aggregating the nodes’ information more precisely from different graphs. Furthermore, we also performed experiments with three kinds of single aggregators. As shown in Tab.2, the model performs the best when three aggregators’ weights are equal. That means three aggregators are equally crucial in alleviating information confusion when aggregating node information. As for the single aggregator, the model with the softmax aggregator does the best for the softmax aggregator can allow an asymmetric message passing in the direction of the strongest signal [19].

As shown in Tab.3, the results of GCN [16], GAT [17] are reported by [14], and we reproduced the result of PATCHY-Diff [18]. The results of other compared methods are reported in their published papers. Our proposed WMA-GIN outperforms all compared graph-based methods, especially on WA. Compared with LSTM-GIN [14] which also adopts GIN structure, WMA-GIN achieves the improvements by 12.1% and 3.3% on WA and UA, respectively. Without applying FA, the performance decreases. It proves that the FA layer can effectively alleviate the information compression problem for the SER task. The results in Tab.3 also indicate that adopting the Multi-stage Loss, assigning different weights to the features at different stages of the network has a positive influence. And the log-Mel spectrogram features are inferior to the LLDs. Furthermore, we also compare six recent advanced methods which do not adopt any GNN structures as shown in Table 4. WMA-GIN utilizing LLDs features outperforms all listed methods on WA.