Adaptive speech emotion representation learning based on dynamic graph

Let $\mathcal{G}=\left\{G^{1},G^{2},\cdots,G^{S}\right\}$ indicate a dynamic graph, different from the traditional methods that only construct a static graph for an entire sequence. $G^{s}$ is an observed graph specific to an audio segment $s$ $\left(1\leq s\leq S\right)$ and the structure of $G^{s}$ varies with different audio segments. $\mathcal{A}=\left\{\mathbf{A}^{1},\mathbf{A}^{2},\cdots,\mathbf{A}^{S}\right\}$ denote the structure of $\mathcal{G}$. More specifically, $G^{s}=\left\{\mathcal{V}^{s},\mathcal{E}^{s},\mathbf{X}^{s}\right\}$, where $\mathcal{V}^{s}=\left\{v_{1},v_{2},\cdots,v_{m}\right\}$ indicate that each audio segment $s$ has $m$ nodes, $\mathcal{E}^{s}$ is an edge set at audio segment $s$, and $\mathbf{X}^{s}$ is a feature matrix for all nodes at audio segment $s$. Given a dynamic graph $\mathcal{G}$, the goal of graph representation learning is to study a function $f:\mathbb{R}^{m\times m}\to\mathbb{R}^{m\times q}$ for each $G^{s}$ in $\mathcal{G}$. Specifically, based on the function $f$, the given $\mathcal{G}$ can be output as low-dimensional representations, e.g., $\mathcal{H}=\left\{\mathbf{H}^{1},\mathbf{H}^{2},\cdots,\mathbf{H}^{S}\right\}$, where $\mathbf{H}^{s}\in\mathbb{R}^{m\times q}$.

Our graph construction follows the frame-to-node transformation, as shown in Fig. 2.

A frame represents a 25ms audio. To encode the temporal information, neighboring nodes (frames) need to be connected. Meanwhile, to aggregate the information of distant nodes, we also randomly connect the nodes. The $a_{ij}\in\mathbf{A}^{s}$ represents the weight corresponding to the edge $e_{ij}\in\mathcal{E}^{s}$ between $v_{i}$ and $v_{j}$ nodes. Note that the graph structure is not naturally defined here, i.e., the elements in $\mathbf{A}$ are unknown. The common methods to determine the $\mathbf{A}$ include cosine similarity function [1], manual definition [2], and distance function [16]. However, these may lead to a sub-optimal graph [17]. Therefore, we propose a new matrix method to initially the $\mathbf{A}$ and optimize the graph structure during the training by combining graph structure loss and classification loss. This loss function will be presented in Subsection 3.3.

Calculating the similarity between each node and its neighboring nodes is crucial in graph-based analysis. Dice similarity is a well-known node similarity measurement function [18]. Since Dice similarity is calculated directly based on the network topology, it is relatively interpretable and saves computing resources. Given two nodes $v_{i}$ and $v_{j}$, the Dice similarity score is calculated as follows:

$$\mathbf{S}_{Dice}\left(v_{i},v_{j}\right)=\frac{2Con\left(v_{i},v_{j}\right)}{N\left(v_{i}\right)+N\left(v_{j}\right)},$$

(1)

where $Con\left(v_{i},v_{j}\right)$ indicates the number of common neighbors. $N\left(v_{i}\right)$ and $N\left(v_{j}\right)$ denote the number of neighbors of $v_{i}$ and $v_{j}$ (not contain the node itself), respectively. As can be seen from Eq. (1), when calculating the similarity between the nodes, the degree of neighboring nodes does not affect the final result.

However, nodes with higher degrees are more likely to have more influence and higher weight values [19]. For example, in the social networks of NBA players, players with more followers may bring more attention to their teams than players with similar physical conditions and skills. Take Fig. 3 as an example,

the common neighbors number of node $v_{2}$ with node $v_{1}$ is equal to the node $v_{3}$ with node $v_{1}$. The influence of node $v_{2}$ with smaller degree is $\mathbf{S}_{Dice}\left(v_{2},v_{1}\right)=0.33$. However, the influence of node $v_{3}$ with larger degree is $\mathbf{S}_{Dice}\left(v_{3},v_{1}\right)=0.29$, which is not expected in the real world. This is because graph-based methods usually update node representations by transferring and aggregating information from neighboring nodes. Thus, the higher-degree nodes may aggregate more information.

Therefore, we propose a new Dice matrix calculation method to solve the problem that the neighboring node degree has no positive effect on the target node. The proposed method is defined as follows:

$$Con\left(v_{i},v_{j}\right)=N\left(v_{i}\right)\cap N\left(v_{j}\right),$$

(2)

$$D\left(v_{i},v_{j}\right)=\frac{\left|N\left(v_{i}\right)\cup\left\{v_{i}\right\}\right|}{\left|N\left(v_{i}\right)\cup\left\{v_{i}\right\}\right|+\left|N\left(v_{j}\right)\cup\left\{v_{j}\right\}\right|},$$

(3)

$$\mathbf{A}_{Dice}^{s}\left(v_{i},v_{j}\right)=\frac{Con\left(v_{i},v_{j}\right)+D\left(v_{i},v_{j}\right)}{\left|N\left(v_{j}\right)\cup\left\{v_{j}\right\}\right|+1},$$

(4)

where $Con\left(v_{i},v_{j}\right)$ represents the number of neighbors shared by node $v_{i}$ and node $v_{j}$, and $D\left(v_{i},v_{j}\right)$ indicates the degree of neighbor node $v_{i}$ (the node itself is contained). According to Eq. (4), as expected, the influence of node $v_{2}$ on $v_{1}$ and the influence of node $v_{3}$ on $v_{1}$ in Fig. 3 are $\mathbf{A}_{Dice}^{s}\left(v_{2},v_{1}\right)=0.56$ and $\mathbf{A}_{Dice}^{s}\left(v_{3},v_{1}\right)=0.57$. That is to say, the nodes with relatively large degrees have higher node weight values.

Traditional GCN layer usually takes the node feature matrix $\mathbf{X}\in\mathbb{R}^{m\times p}$ and the graph adjacency matrix $\mathbf{A}\in\mathbb{R}^{m\times m}$ as inputs to generate the node-level representation matrix $\mathbf{Z}\in\mathbb{R}^{m\times q}$. The GCN layer can be described as:

$$\mathbf{H}^{\left(l+1\right)}=\sigma\left(\hat{\mathbf{A}}\mathbf{H}^{\left(l\right)}\mathbf{W}^{\left(l\right)}\right),$$

(5)

where $\sigma$ is a $ReLU$ activation function that implements nonlinearity. $\hat{\mathbf{A}}=\mathbf{D}^{-\frac{1}{2}}\left(\mathbf{A}+\mathbf{I}\right)\mathbf{D}^{-\frac{1}{2}}$, $\mathbf{D}$ is a the degree matrix of $\mathbf{A}$, and $\mathbf{I}$ is a $m\times m$ identity matrix. $\mathbf{H}^{\left(0\right)}=\mathbf{X}$, $\mathbf{H}^{\left(L\right)}=\mathbf{Z}$, and $\mathbf{W}^{\left(l\right)}$ is the weight matrix for the $l^{th}$ layer.

We present a new graph convolution structure for SER. It consists of the following two novel components:

$\bullet$ Segment-specific graph convolution. For each audio segment $s$, we generate a node representation matrix $\mathbf{H}^{s}$. The graph convolution layer is represented as:

$$\mathbf{H}^{s}=\sigma\left(\hat{\mathbf{D}}_{s}^{-\frac{1}{2}}{\mathbf{A}}^{s}\hat{\mathbf{D}}_{s}^{-\frac{1}{2}}\mathbf{X}^{s}\mathbf{W}^{s}\right),$$

(6)

where $\hat{\mathbf{D}}$ is a degree matrix of ${\mathbf{A}}^{s}$ and ${\mathbf{A}}^{s}=\tilde{\mathbf{A}}^{s}+\varphi\mathbf{A}_{Dice}^{s}+\mathbf{I}$. The $\tilde{\mathbf{A}}^{s}_{i,j}=\left(i-j\right)^{2}$, $\mathbf{A}_{Dice}^{s}$ is a novel matrix calculated by the proposed Eq. (4), and $\mathbf{I}$ is an identity matrix. $\mathbf{X}^{s}$ is a feature matrix and $\mathbf{W}^{s}$ is a weight matrix, where $\mathbf{W}^{0}$ is obtained by random and $\mathbf{W}^{s+1}=\sigma\left(\mathbf{W}^{s}+{\mathbf{A}}^{s}*\mathbf{X}^{s}\right)$. Different from the traditional GCN that employs $\mathbf{A}+\mathbf{I}$ to guide the aggregate of node representations. We additionally add the $\mathbf{A}_{Dice}$ to further guide the aggregation and use a parameter $\varphi=0.6$ to control its contribution. Under the guidance of the new aggregation strategy, a higher-quality node representation can be obtained.

$\bullet$ Learnable adjacency matrix ($\mathbf{A}_{Learn}$). We combine graph classification loss ($\mathcal{L}_{GC}$) and graph structure loss ($\mathcal{L}_{GL}$) to jointly optimize the graph structure during training. The $\mathcal{L}_{GC}$ is defined using the cross-entropy loss:

$$\mathcal{L}_{GC}=-\sum_{n=1}^{N}y_{n}\log{\hat{y}_{n}},$$

(7)

where $\hat{y}_{n}$ represents the prediction label for the $n$-$th$ sample. The $\mathcal{L}_{GL}$ is implemented as follows:

$$\mathcal{L}_{GL}=\underset{\text{graph structure loss}}{\underbrace{\lambda_{1}e^{T}\left(\mathbf{A}_{Dice}\odot\mathbf{A}_{Learn}\right)e+\lambda_{2}\left\|\mathbf{A}_{Dice}\right\|_{F}^{2}}},$$

(8)

where $\lambda_{1}$ and $\lambda_{2}$ control the proportion of these items respectively, $e$ is an all-ones vector, $\left\|\cdot\right\|_{F}$ indicates Frobenious norm, and $\mathbf{A}_{Dice}$ refers to Eq. (4). The overall loss function is as follows:

$$\min_{A_{l},p,\Theta}\mathcal{L}=\min_{A_{l},p,\Theta}\left[\mathcal{L}_{GC}+\mathcal{L}_{GL}\right],$$

(9)

where $\Theta$ represents all learnable parameters on all graph convolution layers. Each term in the overall loss function $\mathcal{L}$ is differentiable, thus allowing end-to-end optimization.

The IEMOCAP database contains 12 hours of data. To keep consistent with previous studies, only four emotions are used. We utilize the OpenSMILE tool to extract features from the audio. For each sample, we use a sliding window of length 25ms (with a step length of 10ms) to locally extract the low-level descriptors (LLDs), such as signal strength, power spectral density, and base frequency. Each speech sample contains 120 nodes, where each node represents an (overlapping) audio frame of length 25ms. The RAVDESS database includes 1,500 speech samples, performed by 24 professional actors (12 females and 12 males). Each actor simulated eight different emotional states. We use the Fourier transform to convert the speech signal into frequency domain representations. Then, we extract frequency domain features, such as the Mel frequency cepstrum coefficient (MFCC), where the sampling rate is 22,050 Hz and the number of MFCCs is 40. Each audio sample contains 40 nodes, where the node corresponds to a 25ms audio frame. Appropriately increasing the number of MFCCs can help perceive changes in sound signals at different frequencies, and these detailed features may play a positive role in emotional sensitivity. The reason for extracting MFCC features from the RAVDESS dataset is that it provides representative spectral information.

we trained the model to have up to 1000 epochs, with 150 iterations per epoch, and used an early stop strategy. The batch size of the model was set to 64. Meantime, the RAdam optimizer with a learning rate of 0.001 was employed and we set the decay rate to 0.5 after every 150 epochs. We conducted experiments on GeForce GTX 3090 Ti, NVIDIA-SMI 460.39, and CUDA Version 11.2 GPUs, and obtained the final experimental data by using the 10-fold cross-validation averaging method.

Comparison with SOTA methods. Table 1(b) presents all results on two datasets. On the IEMOCAP, compared with the graph-based SOTA approaches, the performance of our model seems only slightly better (1.44% of accuracy absolutely) than L-GrIN [3], but the adjacency matrix $\mathbf{A}$ of L-GrIN is obtained by a distance function, which tends to locally optimal graphs during the training. When compared with other SOTA non-graph methods, we find that our method outperforms popular Transformer-based methods. This is mainly attributed to the fact that we connect neighboring nodes while randomly connecting distant nodes for information transfer. The relationship between the nodes is calculated by the proposed matrix calculation method to guide the aggregation of node information, thus capturing the long-distance information. Table 1(b)(b) also shows that our model performs better than the graph-based and non-graph-based approaches on the RAVDESS database. For example, compared with the classic GCN model, our model has great performance advantages (15.83% of accuracy absolutely). This indicates that the proposed matrix computation method can alleviate the sub-optimal graph in traditional GCN caused by averaging the information of neighboring nodes.

Ablation experiment. We perform ablation studies on two datasets, including the following variations of our model: Binary: manually defining the structure of the graph, the matrix A only contains $0$ and $1$; Weighted: the relationship between nodes is determined based on the distance; $\mathbf{A}_{Learn}$: the structure of the graph is learned during the training; $\mathbf{A}_{Dice}$: the loss function only contains the proposed matrix computation method. The results of the study are displayed in the “Adjacency matrix” of Table 1(b). We have the following observations.

Firstly, both Binary and Weighted determine the relationship between nodes through customized form, which makes them easy to form the local optimal graphs during the training process, so the performance is the worst. Secondly, the proposed matrix calculation method $\mathbf{A}_{Dice}$ has enhanced performance compared to the learnable $\mathbf{A}_{Learn}$, which shows that our method can effectively distinguish the significance of different neighboring nodes. Finally, we optimize the structure of the graph by jointing the graph structure loss and classification loss to achieve the best performance. This indicates that the proposed matrix computation method can update the node representation and that the structure of the graph can be continuously optimized during the training process.