Mental-Perceiver: Audio-Textual Multimodal Learning for Mental Health Assessment

Early studies of automatic anxiety/depression detection were concentrated on extracting effective features from the responses to questions that were highly correlated with anxiety/depression. Williamson et al. (Williamson et al. 2016) used related semantic context cues entailed in the voice, video-based facial action units, and transcribed text of individuals and built a Gaussian Staircase Model to detect depression automatically. Yang et al. (Yang et al. 2016) selected depression-related questions after analyzing interview transcripts manually and constructed a decision tree with the selected questions to predict the participants’ depression states. Similarly, Sun et al. (Sun et al. 2017) first manually selected questions related to certain topics such as recent feelings and sleep quality by conducting content analysis based on the text transcripts of clinical interviews. Then, they extracted text features from these selected questions as the input of Random Forest to detect depression tendencies. Gong et al. (Gong and Poellabauer 2017) performed topic modeling to split the interviews into topic-related segments. Then, they maintained the most discriminating features by a feature selection algorithm. Giannakakis et al. (Giannakakis et al. 2017) focused on non-voluntary and semi-voluntary facial cues to estimate the emotion representation more objectively. They selected the most robust features from the features including eye-related events, mouth activity, head motion parameters, and heart rate estimated through camera-based photoplethysmography. Finally, they deployed a ranking transformation to investigate the correlation of facial parameters with a participant’s perceived amount of stress/anxiety.

With the advancement of deep learning, extracting and integrating multi-modal features through deep learning models have become particularly promising for anxiety/depression detection. Ma et al. (Ma et al. 2016) encoded the depressive audio characteristics with the combination of CNN and LSTM to predict the presence of depression. Yang et al. (Yang et al. 2017) trained a deep Convolution Neural Network (CNN)-based depression detection model with specially-designed a set of audio and video descriptors. Tuka et al. (Al Hanai, Ghassemi, and Glass 2018) selected audio features and text features that were strongly related to depression severity by computing Pearson Coefficients and built a Long Short-Term Memory (LSTM) network to assess depression tendency. Haque et al. (Haque et al. 2018) proposed a causal CNN model to summarize acoustic, visual, and linguistic features into embeddings which were then used to predict depressive states. Shen et al. (Shen, Yang, and Lin 2022) proposed a depression detection approach utilizing speech characteristics and linguistic contents from participants’ interviews. Lin et al. (Lin et al. 2022) used the biological information of speech, combined with deep learning to build a rapid binary classification model of depression in the elderly. Agarwal (Agarwal, Jindal, and Singh 2023) developed machine-learning solutions to diagnose anxiety disorders from audio journals of patients.

The public anxiety/depression datasets are quite scarce (Gratch et al. 2014; Valstar et al. 2014; Shen, Yang, and Lin 2022) due to ethical issues. To the best of our knowledge, there are only three publicly available datasets  (Gratch et al. 2014; Valstar et al. 2014; Shen, Yang, and Lin 2022) referring to depression detection while there are no publicly available multi-modal datasets including audio, facial, or text referring to anxiety detection. The DAIC-WoZ dataset (Gratch et al. 2014) contains recordings and transcripts of 142 American participants who were clinically interviewed by a computer agent automatically. AVid-Corpus (Valstar et al. 2014) contains audios and videos of German participants answering a set of queries or reciting fables. However, the text transcripts are not provided. EATD-Corpus (Shen, Yang, and Lin 2022) is released to facilitate the research in depression detection. It consists of audios and text transcripts extracted from the interviews of 162 student volunteers recruited from Tongji University. Although EATD-Corpus provides the text transcripts, its data scale is small.

Following the pioneering work PerceiverIO (Jaegle et al. 2021), all attention modules deployed in our Mental-Perceiver are implemented as Transformer-style attention (Vaswani et al. 2017). Each attention module applies a global query-key-value (QKV) attention operation followed by a multi-layer perceptron (MLP). The MLP is independently applied to each element of the index dimension. Both the encoder and decoder take in two input feature arrays. The first array is used as input to the attention module’s key and value networks, and another array is used as input to the module’s query network for interacting and fusing with the first array. The attention module’s output has the same index dimension (the same number of elements) as the query input. Therefore, the attention module can be modeled as follows:

$$\text{attention}(Q,K,V)=MLP(Softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V)$$

(1)

Our Mental-Perceiver extracts deep features based on the latent features $z$ which is produced by fusing the multimodal input $x$ and the latent embedding $p$ with an attention module. $p$ is used as a query and $x$ is used as the key and value. This means that the output $z$ is the result that $p$ extracts and fuse semantic information from $x$. So, the initialization of the latent embedding $p$ is crucial for learning more discriminative features in the following steps to identify a user’s mental state according to the user’s audio and transcript text. Semantic prior has been shown to help greatly learn more discriminative features (Teney, Abbasnejad, and Hengel 2019; Dai et al. 2023; Ding et al. 2023). Therefore, we build semantic priors for different categories, one for the normal category and another for the mental disorder category. Then, we use these two semantic priors to initialize the latent embedding $p\in\mathbb{R}^{2\times D}$ with a learnable MLP for mapping the hidden size of semantic priors to $D$. In Mental-Perceiver, we fixed the parameters of these two semantic priors and only optimized the parameters of the learnable MLP semantic priors. To obtain the semantic priors, we simply compute the center points for different categories.

Formally, given the text representation set $\textbf{E}_{t}\in\mathbb{R}^{N\times H}$ of a category on the training set, where $N$ is the number of data samples for the current category and $H$ is the hidden size of a text representation, we can compute the semantic prior $p^{C_{i}}\in\mathbb{R}^{1\times H}$ by averaging the normalized $\textbf{E}_{t}$ at the index dimension. This procedure can be modeled as follows:

$$p^{C_{i}}=Avg(Norm(\textbf{E}_{t}))$$

(2)

where $Avg$ is the averaging function and $Norm$ is the z-score normalization function that normalizes each vector separately in $\textbf{E}_{t}$. Let $C_{0}$ and $C_{1}$ denote the classes of normal people and psychological patients, respectively, we first obtain two semantic priors $p^{C_{0}}$ and $p^{C_{1}}$ according to Equation (2). Then, we obtain $p$ by concatenating $p^{C_{0}}$ and $p^{C_{1}}$ at the index dimension followed by a learnable MLP layer as follows:

$$p=MLP(p^{C_{0}}\odot p^{C_{1}})$$

(3)

where $\odot$ denotes the concatenation.

The encoder, consisting of an attention module, takes charge of mapping the multimodal input $x$ into latent features $z$ by utilizing category semantic prior-enhanced latent embedding $p$ to query the multimodal input $x$. This way will fuse the multimodal input $x$ and category semantic prior-enhanced latent embedding $p$ and build fused prior-guided latent features $z$, which will be used as input to the next deep feature extraction. The encoder can be modeled as follows:

$$z=attention(p,x,x)$$

(4)

Once we obtain the latent features $z$, we conduct deep feature extraction based on the input latents $z$ by applying a series of attention modules that take in and return latents $z_{i}$ in this latent space iteratively.

This module can be modeled as follows:

	$\displaystyle z_{1}$	$\displaystyle=attention(z,z,z)$		(5)
	$\displaystyle z_{2}$	$\displaystyle=attention(z_{1},z_{1},z_{1})$
		$\displaystyle...$
	$\displaystyle z_{k}$	$\displaystyle=attention(z_{k-1},z_{k-1},z_{k-1})$

where $k$ is a hyper-parameter that is simply set to 8.

The goal of the decoder is to produce a final class-wise logit output of size $2\times 2$, given a latent representation of size $2\times D_{z}$. Let $z^{\prime}=z_{k}$, the decoder first applies an attention module to map the latent $z^{\prime}$ to output features $y$. Then, the decoder applies a linear layer to map $y$ into the final class-wise logit output $[y^{C_{0}},y^{C_{1}}]\in\mathbb{R}^{2\times 2}$, where $y^{C_{0}}\in\mathbb{R}^{1\times 2}$ and $y^{C_{1}}\in\mathbb{R}^{1\times 2}$ are two logit outputs for indicating whether the multimodal input $x$ matches with semantic prior $p^{C_{0}}$ and $p^{C_{1}}$. Finally, we compute the mean vector of these two output logit vectors $y^{C_{0}}$ and $y^{C_{1}}$ at the index dimension as the final classification logits $y^{\prime}$. Therefore, the decoder can be modeled as follows:

	$\displaystyle y$	$\displaystyle=attention(q,z^{\prime},z^{\prime})$		(6)
	$\displaystyle[y^{C_{0}},y^{C_{1}}]$	$\displaystyle=Linear(y)$
	$\displaystyle y^{\prime}$	$\displaystyle=Mean(y^{C_{0}}\odot y^{C_{1}})$

To optimize our Mental-Perceiver, we deploy two losses. The first one is the matching loss $\mathcal{L}_{match}$ and another one is the classification loss $\mathcal{L}_{cls}$. Both these two losses are binary cross-entropy loss functions. The matching loss aims to optimize the matching degree between the multimodal input $x$ and its corresponding category semantic prior while the classification loss optimizes the model to be able to identify the inherent mental state according to multimodal input $x$ with the help of category prior $p$ and category query $q$.

Formally, given a multimodal input $x$ and its class $C_{x}$ which can be 0 or 1, the training objectives can be modeled as follows:

	$\displaystyle\mathcal{L}_{match}=$	$\displaystyle-C_{x}(logy^{C_{0}}_{0}+logy^{C_{1}}_{1})$		(7)
		$\displaystyle-(1-C_{x})(logy^{C_{0}}_{1}+logy^{C_{1}}_{0})$
	$\displaystyle\mathcal{L}_{cls}=$	$\displaystyle-C_{x}logy^{\prime}_{0}-(1-C_{x})logy^{\prime}_{1}$
	$\displaystyle\mathcal{L}=$	$\displaystyle\mathcal{L}_{match}+\mathcal{L}_{cls}$

where $y^{C_{0}}_{0}$ and $y^{C_{0}}_{1}$ are the 0-th element and 1-th element in the probability distribution obtained from class-wise logits $y^{C_{0}}\in\mathbb{R}^{1\times 2}$ by Softmax function while $y^{C_{1}}_{0}$ and $y^{C_{1}}_{1}$ are the 0-th element and 1-th element in the probability distribution obtained from class-wise logits $y^{C_{1}}\in\mathbb{R}^{1\times 2}$ by Softmax function. Similarly, $y^{\prime}_{0}$ and $y^{\prime}_{1}$ are the two probabilities on normal class (0) and mental disorder class (1) that can be obtained by applying the Softmax function on $y^{\prime}$.

We conduct experiments on both two subsets of MMPsy for anxiety detection and depression detection. We use MMPsy-Anxiety and MMPsy-Depression to represent these two subsets. Besides, we also verify our Mental-Perceiver on the Distress Analysis Interview Corpus - Wizard of Oz (DAIC-WOZ) dataset (Gratch et al. 2014). DAIC-WOZ contains clinical interviews of 189 participants designed to support the diagnosis of psychological distress conditions such as anxiety, depression, and posttraumatic stress disorder (PTSD). During each interview, several data in different formats as well as modalities are recorded simultaneously. However, only the acoustic recordings and transcriptions are chosen in this work for a fair comparison. Moreover, the given GT is an eight-item Patient Health Questionnaire depression scale (PHQ-8), which indicates the severity of depression. A PHQ-8 Score $\geq$ 10 implies that the participant is undergoing a mental disorder.

The main baselines to be compared are listed as follows:

• •

  SVM (Pedregosa et al. 2011): a robust shallow model capable of performing binary classification efficiently by using a kernel trick, representing the data only through a set of pairwise similarity comparisons between the original data points using a kernel function, which transforms them into coordinates in the higher dimensional feature space.

• •

  RandomForest (Pedregosa et al. 2011): it is a robust ensemble learning method for classification by constructing a multitude of decision trees at training time. For classification tasks, the output of the random forest is the class selected by most trees.

• •

  XGBoost (Chen and Guestrin 2016): it is a robust toolbox for classification via an optimized distributed gradient boosting.

• •

  NUSD (Wang, Ravi, and Alwan 2023): it is a deep model ECAPA-TDNN enhanced with a speaker disentanglement method that utilizes a non-uniform mechanism of adversarial SID loss maximization.

• •

  ConvLSTM (Wei et al. 2022): it is a Convolutional Bidirectional LSTM with a sub-attention mechanism for linking heterogeneous information.

• •

  PerceiverIO (Jaegle et al. 2021): it is a general-purpose architecture that handles multimodal data with fully attention design and a flexible querying mechanism.

For the shallow models SVM, RandomForest, and XGBoost, we provide the following audio features as input: F0 statistics (mean), log-energy, zero-crossing-rate, loudness, pitch period entropy, jitters, shimmers, harmonics-to-noise ratio, detrended fluctuation analysis, linear spectral coefficients-0, linear spectral frequencies-0, formants (F1), and amplitude Shannon entropy. All these features can be extracted by applying Surfboard ¹¹1https://github.com/novoic/surfboard (Lenain et al. 2020). Besides, we also extract topic words by TF-IDF as text features for these shallow models. For the deep models, we use BERT (Devlin et al. 2019) to extract features and use Mel-spectrum to represent audio.

In classification tasks, True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN) from the confusion matrix are metrics used to measure the accuracy of model predictions. TP refers to instances where the model correctly predicted them as the positive class, while TN refers to instances where the model correctly predicted them as the negative class. On the contrary, FP represents cases where the model incorrectly labeled instances as positive, and FN denotes instances of the positive class that were incorrectly classified as negative.

Based on these concepts, several common performance metrics can be defined to assess the model’s performance:

• •

  Accuracy: Accuracy (Acc) represents the proportion of all predictions that are correctly predicted.

• •

  Recall: Recall measures the model’s ability to identify all true positive instances, that is, the proportion of actual positives that are correctly identified.

• •

  Precision: Precision denotes the proportion of samples predicted as positive by the model that are actually positive, focusing on the accuracy of positive predictions.

• •

  F1-Score: F1 Score ranges from 1 to 0, with a value closer to 1 indicating better model performance, particularly in scenarios where the distribution of positive and negative samples is imbalanced. The F1 Score serves as a more comprehensive evaluation metric under such conditions.

• •

  Sensitivity: Sensitivity (Sens), also termed true positive rate, is the ratio of positive predictions to the number of actual positives. It is hence identical to the recall of the positive class, Recall(1).

• •

  Specificity: Specificity (Spec) is the ratio of negative predictions to the number of actual negatives, and therefore identical to Recall(0).

• •

  UAR: Although Acc is often used to evaluate model performance, it suffers from the data imbalance issue. The stronger the imbalance the more accuracy tends to reflect the performance of the majority class since it is a weighted accuracy. For this reason in areas where relatively small data sets predominate, such as the bio-medical field or in the paralinguistics field in general, researchers usually prefer and report the Unweighted Average Recall (UAR), which is defined as the average of Sensitivity and Specificity.

  $$UAR=\frac{Sensitivity+Specificity}{2}$$

  (8)

During these metrics, we use Accuracy (Acc), Unweighted Average Recall (UAR), Sensitivity (Sens), and Specificity (Spec) as the main evaluation metrics for evaluating overall model performance. Meanwhile, we also report the Recall, Precision, and F1-score separately for different categories.

We use Pytorch²²2http://pytorch.org to implement our framework on Linux with two NVIDIA RTX 4090 GPU cards. The feature dimension $D_{x}$ is set to 768 and other dimensions $D_{z}$, $D_{q}$, and $D_{y}$ are all set to 512. In each epoch, all training data is shuffled randomly and then cut into mini-batches. The text feature and audio feature are concatenated as the multimodal input. We deploy the AdamW (Loshchilov and Hutter 2017) optimizer for model optimization. We trained models for 200 epochs with an initial learning rate of 0.00003 and used LambdaLR to adjust the learning rate during training. The early stopping with patience 15 is deployed to accelerate training. We use the validation set for model selection and report the performance on the test set.