A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning

The emotion recognition method based on the dictionary is the earliest used for emotion recognition (Hardeniya and Borikar, 2016), which motivated early work on naive bayes method (Frank et al., 2000). With the widespread application of machine learning algorithms in classification tasks, the representative machine learning classification algorithm (SVM (Rozgić et al., 2012), (Hu et al., 2007), and binary decision tree (Cichosz and Slot, 2007), (Lee et al., 2011), (Liu et al., 2018b), etc) has also begun to shine in the field of emotion recognition. The above-mentioned method determines the category of emotion by learning the polarity and occurrence frequency of emotional words in the text, which is difficult to extract the semantic information and context information.

Encouraged by the success of CNNs in the field of computer vision (CV), CNNs began to be migrated to text classification tasks and received extensive research attention (Khare and Bajaj, 2020), (Kwon et al., 2021), (Kollias and Zafeiriou, 2020). In 2017, Poria et al. (Poria et al., 2017) used LSTM for the first time to resolve dependencies between contexts. Since then, improvements, extensions and applications of LSTMs and GRUs have increased (Hazarika et al., 2018b), (Majumder et al., 2019), (Hazarika et al., 2018a), (Rajamani et al., 2021). Until recently, many GNN-based methods (e.g., (Ghosal et al., 2019), (Ishiwatari et al., 2020), (Shen et al., 2021), (Li et al., 2023), (Zadeh et al., 2018b)) emerged. Apart from CNNs, RNNs, and GNNs, many alternative Transformer-based methods (e.g., (Zhong et al., 2019), (Li et al., 2020), (Zhu et al., 2021)) have been developed in the past decades. We detail the categories to which these algorithms belong in Section 4.

MCER methods based on traditional machine learning (Rozgić et al., 2012), (Cichosz and Slot, 2007), (Lee et al., 2011), (Liu et al., 2018b), (Lin and Wei, 2005), (Hu et al., 2007) are closely related to hand-extracted features, which have attracted increasing attention from the data mining and emotion recognition communities. These methods aim to learn the feature embeddings of raw data for subsequent downstream tasks such as classification and clustering. The classic conversational emotion recognition method based on machine learning is to use support vector machine to map emotional features to a hyperplane and classify them (Rozgić et al., 2012), (Bhavan et al., 2019). However, these methods require a large amount of high-quality labeled data.

The CNN-based emotion recognition methods (Lu et al., 2023), (Kwon et al., 2021), (Kollias and Zafeiriou, 2020) are the first deep learning method to solve the emotion classification problem historically (Kim, 2014). These CNN-based methods employ convolutional filters to extract semantic features of text so that the model can use supervised learning to understand the meaning of text. Similar to machine learning algorithms, CNN can also map emotional features into vector space through mapping functions. The difference is that this mapping function is learned in an end-to-end manner. Since the convolution kernel extracts local receptive field information, CNN cannot contain contextual semantic information.

The RNN-based emotion recognition methods (Ma et al., 2019), (Tao and Liu, 2018), (Majumder et al., 2019), (Kollias and Zafeiriou, 2020) are developed on the basis of CNN, but they believe that contexts should be mutually influential and interdependent (Ma et al., 2019), (Tao and Liu, 2018). These RNN-based methods usually use LSTM or GRU (to avoid gradient disappearance or gradient explosion) to extract semantic features including context. Similar to CNN, RNN can also map emotional features into vector space through mapping functions in an end-to-end manner.

Similar to the RNN-based emotion recognition method, the Transformer-based emotion recognition methods (Huang et al., 2020), (Lian et al., 2021), (Tsai et al., 2019), (Rahman et al., 2020) also extract semantic information including context, and completes subsequent emotion classification based on this (Lian et al., 2021). However, unlike RNN, Transformer’s sequential context modeling ability is better than RNN. Therefore, the accuracy of the Transformer-based emotion recognition methods are significantly better than RNNs.

The GNN-based emotion recognition methods (Lin et al., 2022), (Zhang et al., 2023a), (Li et al., 2023), (Ishiwatari et al., 2020), (Shen et al., 2021) inherit the idea of the RNN method, i.e., the contexts should interact and depend on each other (Ghosal et al., 2019). On the basis of RNN, GNNs believe that there is also a relationship of mutual influence between speakers. Therefore, GNNs model the dialogue relationship between speakers through the inherent properties of the graph structure.

With the rapid development of deep learning technology, word embedding technology has also been widely used to extract text features. Word embedding technology uses a shallow neural network to learn the semantic information of words, and uses Euclidean distance to measure the similarity between words. Unlike traditional one-hot encoding methods, word embedding technology can map high-dimensional sparse feature vectors to low-dimensional dense feature vectors, thus saving a lot of computing resources and solving the problem that one-hot encoding cannot distinguish the semantic gap between words. A commonly used word embedding method is Word2Vec (Chen et al., 2019), which contains two different forms: CBOW (Ghosh and Anwar, 2021) and Skip-gram (Du et al., 2020). CBOW predicts the central word based on the surrounding words, and Skip-gram predicts the surrounding words based on the central word. Although the above methods can capture the semantic similarity between words, they require large datasets for training.

Some recent studies use simple TextCNN (Kim, 2014) and GLOVE (Gan et al., 2022) to extract text features. In addition, large-scale predictive pre-training models such as BERT (Ma et al., 2021) and RoBERTa (Kim et al., 2021) are often used, which capture contextual information through attention mechanisms.

Visual feature extraction is mainly to extract information such as facial expressions and gestures that contain the speaker’s emotions from the video. In recent years, deep neural networks have been able to extract deep features from images in an end-to-end learning manner, avoiding the tedious manual feature extraction. For example, Tran et al. (Tran et al., 2015) proposed an effective and efficient 3D-CNN to process video frames containing spatio-temporal features.

Nowadays, most of the visual features are extracted by some advanced neural networks (e.g., CNN (Kattenborn et al., 2021; Wang et al., 2020a), and Transformer (Han et al., 2021; Han et al., 2022), etc) or some open source toolkits (e.g., OKAO Vision (Zhu et al., 2023), OpenFace (Baltrusaitis et al., 2018), CERT (Boopathy et al., 2019) and Facet (Barreiro and Treglown, 2020), etc). OKAO Vision extracts the speaker’s facial expression in each frame of the video, and finally gets the speaker’s smile intensity (0-100) and eye gaze direction. CERT can adaptively extract visual features such as face and head pose. OpenFace 2.0 is capable of detecting facial landmarks, recognizing facial movements, and estimating eye gaze direction. Facet extracts visual features, including facial movements, head poses, HOG features and etc.

Deep learning has also begun to get more and more attention in research in audio feature extraction research. For example, LSTM (Xie et al., 2019) has been widely used to automatically extract acoustic characteristics. Poria et al. (Poria et al., 2017) uses CNN to extract features from the audio, and then enter the extracted audio features into the classifier for emotional classification.

Recently, more and more emotion analysis models (Tzinis et al., 2018; Majumder et al., 2019; Zhong et al., 2019) have started to use open source toolkits such as CONVAREP (Dumpala et al., 2023), openSMILE (Kumar et al., 2022), LibROSA (Suman et al., 2022), and OpenEAR (Schepker et al., 2020) to extract audio features. Specifically, OpenEAR adaptively extracts a set of acoustic features (e.g., prosody, spectrum and sepstral, etc.) and uses Z-normalization to normalize the audio features. The features extracted by openSMILE consist of MFCC, pitch and sound intensity. The LibROSA Speech Toolkit extracts 33 frame-level acoustic features (i.e., 20-dimensional MFCC and CQT) including speaker intonation variations. Similar to other audio feature extraction methods, COVAREP can also be used to extract features such as 12-dimensional MFCC, maximum dispersion scale (MDQ) and Liljencrants-Fant (LF).

The early fusion method based on addition operation obtains the final emotional feature representation by weigh-ted summation of different modality features (Deng and Ren, 2021). This fusion method is simple to operate and requires only a small amount of calculation. However, its shortcomings are also obvious. It cannot model the context information in a fine-grained manner, and the information that can be utilized is limited. The formula for implementing the context-free modeling method using the additive approach is defined as follows:

where $h_{e}$ represents the fused emotional vectors, $x^{t},x^{a},x^{v}$ represent the text, audio, and video vectors, respectively.

The early fusion method based on concatenation operation obtains the final emotion feature representation by concatenating and merging different modal features (Cambria et al., 2018). Although this fusion method does not introduce additional calculations, it leads to very high dimensionality of the data, which makes calculations difficult. Furthermore, it also fails to capture intra-modal and inter-modal semantic information that is complementary.

where $Concat\left(\cdot\right)$ represents concatenation operation.

Morency et al. used text, video, and audio features for the task of trimodal emotion analysis, and designed a model to extract useful information in different modal features (Morency et al., 2011). After extracting multi-modal features, the three modal features are connected and input into a Hidden Markov Chain (HMM) classifier (Morency et al., 2011) to learn the emotional state of the input signal. HMM believes that the state of the current moment is only related to the information of the previous moment, which makes the model unable to use the context information of the utterance. The formula of HMM is defined as follows:

where $C,D,M$ represent feature vector dimensions for audio, video, and text, $w$ represents the emotional class, $P(\lambda_{i}^{a},\lambda_{j}^{v},\lambda_{k}^{t}|x^{a},x^{v},x^{t})$ represents the confidence of the emotion classification.

Since the true class label is based on the output of the predicted class label $\hat{w}_{b}$, the formula of HMM can be expanded as follows:

where $P(\hat{w}_{b}|\lambda_{i}^{a},\lambda_{j}^{v},\lambda_{k}^{t},x^{a},x^{v},x^{t})$ represents the probability of predicted label.

SVM is a machine learning algorithm for classification and regression whose optimization goal is to find a hyperplane (a straight line in two-dimensional space, and a hyperplane in high-dimensional space) that separates samples of different classes. Based on the above research, Perez-Rosas et al. (Pérez-Rosas et al., 2013) concatenate multi-modal features as input vectors and use SVM to classify utterances for emotion. SVM works better for binary classification problems, but is less effective in multi-classification problems, and is only suitable for training small-scale data sets. The formula of SVM is defined as follows:

where $sign(x>0)=1,sign(x=0)=0,sign(x<0)=1$, $\alpha_{i}^{*},b^{*}$ represents the learnable parameters, $\exp\left(-\frac{\|x-z\|^{2}}{2\sigma^{2}}\right)$ represents the kernel function, $N$ is the number of the samples.

After preprocessing the features of three different modalities, Poria et al. (Poria et al., 2015) constructed two different feature selectors to achieve feature dimensionality reduction. One of the feature selectors is based on circular correlated feature subset selection (CFS), and the other is based on principal component analysis (PCA). The above two feature selectors can not only eliminate redundant information and noise information, but also improve the running speed of the model. After feature selection and dimensionality reduction, the researchers spliced and merged the processed feature vectors and trained a classifier using a multi-kernel learning (MKL) algorithm (Poria et al., 2015). Based on the previous research work, the authors further propose the Convolutional Recurrent Multi-kernel Learning (CRMKL) (Poria et al., 2016) model. CRNKL uses a convolutional recurrent neural network for emotion detection, which can extract contextual information. The formula of MKL is defined as follows:

where $y_{i}$ is the true label, $\alpha,\beta$ are the learnable parameters, $M$ is the feature dimension.

CNN is a classic neural network in visual tasks and cannot be directly used for emotion recognition. In order to solve this problem, Kim et al. (Kim, 2014) proposed the TextCNN model, and its overall process is shown in Fig. 5. In order to perform multi-modal emotion recognition, Wang et al. (Wang et al., 2017) proposed the SAL-CNN model, which first uses multi-modal data to fully train the CNN, and then uses Select-Additive Learning (SAL) to improve its versatility and prevent the model from overfitting during training. The SAL method consists of two phases (i.e., selection and addition). In the selection phase, SAL preserves important features and removes noisy information from the latent feature representations learned from neurons. In the addition phase, SAL improves the model’s noise immunity by adding Gaussian noise to the feature representation. The SAL method improves the generalization performance of deep fusion models.

The formula for extracting text features by CNN is defined as follows:

where $\oplus$ represents concatenation operator, $\omega$ represents convolution filter, $c_{i}$ represents the feature representation within a window, $f(\cdot)$ represents activation function. Convolutional filters are used to extract features from all sentences to generate feature maps:

The max pooling operation is used to capture the most critical semantic information in the sentence.

It can be seen from the processing flow of the convolutional neural network that using CNN to extract text features does not contain contextual information, i.e., it is assumed that each sentence is independent of each other.

The tensor-based feature fusion method mainly calculates the tensor product of different modal feature representations through Cartesian product to obtain the fused tensor representation (Pandey et al., 2022). Therefore, the above methods need to first map the input multi-modal feature representation into a high-dimensional space, and then map it back to a low-dimensional tensor space for emotion representation. Tensor-based methods are able to capture important high-order interaction information across time, space, and modality. However, the computational complexity of tensor methods is very high and grows exponentially, and there is no fine-grained semantic information interaction between modalities. Zadeh et al. (Zadeh et al., 2017) proposed the multi-modal tensor fusion network TFN. TFN adopts the method of tensor fusion, which can simulate the interaction process between the three modalities of text, audio and video, and effectively fuse multi-modal features. Although TFN can effectively model information interaction within and between modalities, the model complexity of the TFN method is related to the dimensionality of multi-modal features and grows exponentially. The formula of TFN is defined as follows:

where the extra dimension with 1 is used to perform modal interaction. The Cartesian product is then used to fuse the three modal features as follows:

where $\otimes$ represents the outer product, $x^{m}$ represents fused vectors.

On the basis of TFN, in order to more efficiently fuse multi-modal data, Liu et al. (Liu et al., 2018a) proposed a low-rank tensor method LFM to achieve dimensionality reduction of multi-modal features, so as to improve the fusion efficiency of multi-modal features as shown in Fig. 6. LFM has achieved high performance on many different tasks.

where $\mathbf{w}_{a},\mathbf{w}_{v},\mathbf{w}_{t}$ represents the decomposed low-rank learnable tensor.

Multimodal emotion recognition based on adversarial learning is an advanced direction in this field, which combines the principles of adversarial learning to improve the accuracy and robustness of emotion recognition (Ren et al., 2023), (Yuan et al., 2023). Next, we introduce the existing overall process of data augmentation based on adversarial generative networks.

1) Conditional GANs Conditional Generative Adversarial Network (cGAN) (Sun et al., 2023) is a variant of GAN that introduces conditional information to more precisely control the output of the generator. The core idea of cGAN is to pass additional condition information to the generator and discriminator during the generation process, thereby generating specific types of data based on given conditions. The main advantage of cGAN is its ability to precisely control the generation process in order to generate data that meets the conditional information. The optimization goal of cGAN is defined as follows:

where $x$ represents real data, and $y$ represents extra information.

2) Adversarial Autoencoders Adversarial Autoencoder (AAE) (Latif et al., 2020) combines the ideas of Autoencoder and GAN. The main goal of AAE is to make this encoding space more continuous and have better data generation capabilities while learning a compressed representation of data. The training objective function of AAE usually includes two parts: one is the reconstruction error of the autoencoder, which is used to ensure the quality of the encoding, and the other is the GAN loss, which is used to make the encoding distribution more continuous and closer to the real distribution. The formula is defined as follows:

where $p_{z}(z)$ represents the prior distribution.

3) Adversarial Data Augmentation Network Adversarial Data Augmentation Network (ADAN) (Wang et al., 2022) includes the following components: autoencoder $R(E(x))$, auxiliary classifier $C(E(x))$, generator $G(z,y)$ and discriminator $D(h)$. First, ADAN aims to learn a latent representation of the input data x in order to preserve the emotional information in it. Second, it attempts to ensure that the generated latent representation is consistent with the emotional information of the input data by matching the posterior distribution $p(h|z,y)$ with $p(h|x)$. Third, ADAN simultaneously strives to minimize the reconstruction error between the input data $x$ and its reconstructed version $\hat{x}$ to ensure high-quality data reconstruction. The generator $G(z,y)$ accepts a sample $z$ drawn from an M-dimensional Gaussian distribution and a one-hot encoding of the emotion label $y$ as input, and the goal is to generate samples in the latent space such that they are indistinguishable from real samples. The discriminator $D(h)$ is optimized to distinguish whether the latent vector $h$ comes from real data or from the generator.

where $\alpha$ determines the contribution of classification error to model optimization.

The speaker relationship modeling method innovatively introduces graph neural network to capture the speaker’s dialogue relationship information while extracting sequential context information. Taking Fig. 9 as an example, it extracts dialogue relationships between speakers and inter-speaker dependencies by constructing a speaker relationship graph.

GCN extends convolution operations into graph-structured data to extract structural information. GCN performs first-order neighbor information aggregation and spectral domain estimation. The formula of GCN is defined as follows:

where $\boldsymbol{W}^{(l)}$ is the learnable parameters, $\tilde{A}=A+I_{n}$, $I_{n}$ is the identity matrix, $\tilde{\boldsymbol{D}}_{ii}=\sum_{j}\tilde{a}_{ij}$. $\boldsymbol{H}^{(l+1)}$ represents the latent feature representations of layer $l+1$.

The steps to apply GCN to the field of multi-modal emotion recognition are as follows. First, each utterance is represented as a node in the graph, and edge relationships are constructed based on the context between utterances (i.e., if there is a dialogue between utterances, an edge is constructed). We then apply GCN to the constructed dialogue graph for speaker-level information extraction. Through the above process, the model can dynamically learn the correlation between sentences. According to the definition of Equation 27, our formula for aggregating surrounding contextual utterence information is deformed as follows:

where $W_{\theta_{1}}$ and $W_{\theta_{2}}$ are the learnable parameters, $\mathcal{N}_{i}^{r}$ represents the neighbor node under the relationship $r\in\mathcal{R}$.

GAT is a variant of GCN that aggregates surrounding neighbor node features through learnable weights with an attention mechanism. GAT captures the more important node features in the graph by calculating the degree of similarity between nodes. The formula for GAT is defined as follows:

where $\alpha_{ij}$ is the edge weight between node $i$ and node $j$.

Similarly, the formula for using GAT to extract conversational relationships between speakers is defined as follows:

The multi-modal method based on GNN is the current mainstream research, which can consider context information and speaker relationship information simultaneously (Ghosal et al., 2019). In order to jointly learn the three tasks of sequential context information, multi-modal information interaction and multi-task representation, Zhang et al. (Zhang et al., 2023a) designed a multi-modal, multi-task interactive graph attention network (M3GAT) to simultaneously model context dependencies, multi-modal emotional interactions, and speaker dependencies. M3GAT can achieve cross-modal feature interaction, capture of sequential contextual semantic information, and correlation between tasks.

In view of the fact that existing graph fusion methods will cause the model to lose important semantic information and fail to eliminate redundant information in the model, Li et al. (Li et al., 2023) proposed a graph network based on cross-modal feature complementarity, which effectively extracts the speaker’s context and interaction information through multiple hypothesis spaces of the graph. This method eliminates the heterogeneity between modalities and fuses modal information by performing different message aggregation on different nodes and edge relationships in the graph, thereby extracting contextual information and speaker relationship information.

Although existing MCER methods use GCN to model conversational relationships between speakers. In particular, the most competitive methods model the dependence of conversational relations between speakers and the importance between conversational relations by using relational graph attention networks. However, existing GCN-based multimodal conversational emotion recognition methods do not consider conversational relationships and sequential information in contextual relationships. Based on the above problems, Ishiwatari et al. (Ishiwatari et al., 2020) introduced relational position coding in RGAT to provide sequence information. The specific flow chart of RGAT is shown in Fig. 10.

The position encoding formula used by RGAT is defined as follows:

where $PE_{ijr}$ represents the relative position distance between node $i$ under relationship type $r$ and its surrounding neighbor nodes $j$. The maximum relative position distance between nodes is clipped to $p$ or $4$ , which represents the context window size. $\mathcal{N}r(i)$ represents the neighborhood of node $i$ under relationship type $r$. In order to make the position encoding information learnable, FFN is used to obtain position embeddings.