Multi-modal deep learning system for depression and anxiety detection

Depression and anxiety are two of the most common psychiatric disorders that, depending on their severity, can have a profound impact on an individual’s well-being and the quality of life [13, 12, 27, 18, 26]. Thus, it is imperative that treatments for depression and anxiety are prioritized as intervention can greatly improve patient outcomes [6, 25]. Global improvement of anxiety and depression treatment options is estimated to have a direct economic benefit over the period from 2016 to 2030 of $239 billion and $169 billion, respectively [4].

Despite the importance of bettering the treatment pipeline, many barriers remain. One of the primary barriers to effective depression and anxiety treatment is the screening process. Traditional methods for screening have a high burden on clinicians and patients in terms of their ease of administration and scoring, no clear reference standard, and the degree of patient activation and monitoring required [23]. Assessment scales such as the Patient Health Questionnaire (PHQ-8) [20] or Generalized Anxiety Disorder (GAD-7) [29] offer a more quantitative basis for screening.

From another perspective, speech and language are two modalities that form a promising and objective basis for mental health screening. It is well-established that depression and anxiety can alter an individual’s general cognition, with specific biases in their attention and memory [5, 22]. These deficits can manifest in altered acoustic and linguistic dimensions of speech. Some of these include altered rate of speech or increased usage of first-person pronouns [24, 16].

With recent advances in natural language processing and computational power, we now have the ability to collect, measure, and analyze speech data on a larger scale. There is also the rise in popularity of digital platforms such as Amazon Mechanical Turk (mTurk) that has eased the burden of data collection from clinically significant populations [8, 30]. All of this has accelerated development of ML models using speech-based biomarkers for depression and anxiety. These include models that classify anxiety and depression as well as those that predict the severity of these diseases [3, 31, 35]. We build upon the existing literature and extend AudiBERT [31] for the classification of depression and anxiety from speech. Our model incorporates more recent sub-module advances in the architecture and experimental settings. Importantly, we also combine both deep-learned and hand-crafted features to best capture the signal of depression and anxiety that is carried through the acoustic and linguistic properties of speech. We demonstrate that our model achieves better performance on the validation dataset.

Depression and anxiety can present themselves through acoustic and linguistic features of speech [24, 16]. Therefore, our architecture (Figure 1) leverages both of these modalities by parallel representation learning from audio and textual data in addition to representation learning from features hand-crafted by domain experts. Our architecture is inspired by AudiBERT [31].

Working with deep-learned representations of speech can allow for our models to capture more abstract signals in speech that can be used for better depression/anxiety detection. In our work, we use pre-trained speech and language representation models which have been shown to be effective and robust for generating representations of acoustics and text [2, 19].

We utilize Wav2Vec 2.0, one of the best acoustic signal representation models, to learn the features from the speech signal. The output of the Wav2Vec 2.0 base-model, pre-trained on 100k hours of the Vox-Populi dataset [34], is forwarded to a two-layer biLSTM [14, 11] and then to a multi-head attention layer with two heads. Vectors $R1$, outputs of multi-head attention representing acoustic signal, are used jointly with linguistic and hand-crafted features for classification.

Transformers-based architectures [7] have significantly improved language representation, and performance on variety of domain-specific tasks including emotion classification [28]. To represent transcripts of human speech, we select the base model of RoBERTa [19], as one of the best performing language models which can be trained with a single GPU. The output of RoBERTa is forwarded to a two-layer biLSTM, whose output is redirected to a multi-head attention layer with two heads. Vectors $R2$ are outputs of multi-head attention representing the linguistic signal.

There is a rich body of work studying the pathology of depression and anxiety that suggests specific changes in the acoustic, the semantic, and lexico-syntactic content of the speech of those who are suffering from these diseases [24, 16]. We use domain-experts hand-crafted features $R3$ as additional signal. List of those features can be found in the Appendix in Table A.1 and A.2.

Vectors $R1$, $R2$, and $R3$ are concatenated together to create a combined representation embedding of the subject’s speech. This representation is passed through two feedforward layers followed by a binary cross-entropy loss. The architecture classifies between disease and no disease. We train two different models, one for depression and another for anxiety task.

We train and evaluate our models using 5-fold cross-validation with the folds constructed such that there is no overlap between the subjects in the training and test fold. We report the mean of precision, recall and F1-score for each model over the 5 folds. Results are achieved using AdamW optimization with learning rate $lr=3e-5$. We use binary cross-entropy with logits loss from the PyTorch library. The model is trained on T4 Tensor Core GPU with 16 GB RAM.

We use Wav2Vec 2.0 and RoBERTa implementations from the HuggingFace library. Due to memory and architecture constraints with inputting large audio files into Wav2Vec2, we also split the audio samples into consecutive 10 second intervals. The audio was sampled at a rate of 16 000 Hz. Then Wav2Vec2 feature extractor is used to create the input. RoBERTa’s input is a speech transcript generated from the audio via ASR. The text is further transformed by the RoBERTa tokenizer and padded to length of 512. Note, we also add several tokens to the tokenizer corresponding to a set of unfilled and filled pauses in the speech. Pre-trained model weights are not frozen and are fine-tuned for 10 epochs with a batch size of 4 due to GPU memory constraints.

As a baseline, we train a feedforward network using hand-crafted features provided by domain-experts only. The network consists of five linear layers followed by Leaky ReLU activation function [21]. Every layer is twice smaller than the previous one and we use a dropout of 0.2 throughout the network. Network is trained using AdamW optimization with learning rate $lr=3e-4$, batch size of 8, and binary cross-entropy with logits loss. We use the same 5-fold cross-validation process as for the proposed model and we report the mean of precision, recall and F1-score.

The results of our experiments are displayed in Table 1. Examining them in aggregate reveals that our models perform better in predicting no diagnosis, and they are struggling to predict diagnosis. We hypothesize that this is partially a function of the data imbalance that exists within our dataset, as most collected depression and anxiety data comes from individuals with lower scores.

The results show that the inclusion of deep-learned features enriches the representation by adding properties that are not fully captured the hand-crafted features, improving the detection of depression and anxiety. This reflects previous results [31], where the addition of deep-learned features, especially text representation models, improved classification performance for depression.

One of the challenges with developing models for classification of depression and anxiety comes from the distribution of data. In our data and much corpora, a majority of the subjects ware classified with having PHQ-8/GAD-7 scores under 10 leading to class imbalance [33, 10]. Imbalance in classes in training data poses a hurdle in development of robust models [17]. Furthermore, within the classes, the distribution of scores is still uneven. A distribution of PHQ-8/GAD-7 scores is long-tailed and skewed towards lower severity cases. This can lead to issues of within-class imbalance that are difficult to resolve [15].

Interestingly, we also find that depression classification results in higher overall F1-score than anxiety classification. One reason for this was likely due to the data imbalance issue in anxiety samples, which was particularly pronounced as compared to depression (12.8% vs. 25.3% with scores above 9). Another potential reason for this worse performance is that acoustic features in anxiety have been shown to not vary as much with severity as compared with depression [1]. This suggests that anxiety prediction through speech-assessment is a harder task than its corollary in depression.

These findings add to the existing body of work that speech is an appropriate modality for depression and anxiety biomarker development. In particular, using both hand-crafted and deep-learned features maximizes the signal that can be extracted from the speech stream. It also shows how prediction performance for these models is often variable with respect to anxiety/depression severity.

In this work, we present a model for the prediction of anxiety and depression from self-administered speech tasks. Our models extend upon previous work that focuses on classification of depression and anxiety and combines it with a set of hand-crafted features that is able to capture many of the nuanced changes in acoustic and linguistic content of depressed and anxious speech. We find that the proposed model, that combines hand-crafted features with deep-learning speech and language representation, improves classification F1-score of both classes compared to the baseline. The results presented in this paper form a promising basis towards the development of better screening tools for anxiety and depression via speech data.