Avengers Assemble: Amalgamation of Non-Semantic Features for Depression Detection

Here, we discuss various fusion techniques employed in our study for the fusion of NSFs and the downstream modeling employed with the fusion techniques. For a detailed visualization, refer to Figure 1.

We introduce FuSeR, a simple yet effective feature fusion framework that makes use of bilinear pooling that as shown SOTA performance in multimodal tasks [21]. The proposed framework is shown in Figure 1. First, we add convolutional blocks with the same architectural details as used in Early Fusion CNN model on the top of each individual NSFs. After flattening the features, we linear project it to 120 dimension. After this, we apply bilinear pooling that performs outer product on top of the projected features. Mathematically, given two features $\mathbf{a}$ and $\mathbf{b}$, with dimensions $C_{A}\times 1$ and $C_{B}\times 1$ respectively, their outer product yields a matrix $\mathbf{M}$ of dimensions $C_{A}\times C_{B}$, represented as:

$$\mathbf{M}=\mathbf{a}\otimes\mathbf{b}^{T}$$

Here, $\otimes$ denotes the outer product operation, and $\mathbf{b}^{T}$ is the transpose of vector $\mathbf{b}$. The matrix $\mathbf{M}$ encapsulates the pairwise interactions between features from two different NSFs, resulting in a richer representation. The resulting matrix $\mathbf{M}$ is flattened and passed to an FCN module with similar modeling details to the Early Fusion FCN module, followed by an output neuron.

We employ the Extended DAIC (E-DAIC) dataset [22, 23], an enhanced version of the DAIC-WOZ dataset from the AVEC 2019 challenge, which includes semi-structured clinical information consistent with DAIC-WOZ. The E-DAIC dataset comprises data from 275 participants: 163 for training, 56 for development, and 56 for testing, specifically curated to aid in depression diagnosis. This extensive dataset supports robust model evaluation facilitating the development and validation of more effective depression detection methodologies.

We remove the silence and trim the audio to 5 seconds. Further, we resample all the audios to 16kHz before passing it to the PTMs for feature extraction.

We use Mean Squared Error (MSE) as the loss function and Adam as the optimizer. We employ a learning rate of 1e-3 and train all the models for 30 epochs, with a batch size of 64. We use the official split given in the dataset. We trained the models on training set, validated on validation set and tested on testing set. To evaluate model performance comprehensively, we utilize mean absolute error (MAE) and root mean squared error (RMSE) as the evaluation metrics keeping in line with previous research [23, 24]. The training parameters for downstream models with different fusion techniques range between 0.2M to 1.1M.

Firstly, as baseline experiments, we train downstream models on top of the three NSFs considered in our study. We train different downstream models, such as Support Vector Regression (SVR), Random Forest (RF), Fully Connected Network (FCN), and lastly CNN. We use the default parameters as given in Scikit-learn library for SVR and RF. For FCN, we have two dense layers of 120 and 90 neurons, followed by the output that predicts a continuous variable. We follow the same modeling for the CNN model as followed in Section II-B Early Fusion technique. We follow the same training and evaluation regime as the fusion techniques.

The scores for all the tables are scores on the test set. Table I details the performance of different downstream models trained on various NSFs on E-DAIC. Among the NSFs evaluated, TRILLsson with CNN, achieves the lowest RMSE (6.56) and MAE (5.03) scores, indicating superior performance in capturing subtle markers of depression. Conversely, the x-vector features exhibit comparatively higher error rates than the other NSFs.

Table II presents the evaluation results of downstream models trained on different combinations of NSFs, employing various fusion techniques. The results demonstrate that FuSeR, the proposed fusion framework, consistently outperforms the baseline fusion techniques such as Early, Average, and Concatenation methods as well as the individual NSFs. Specifically, when combining x-vector and emoHuBERT, FuSeR achieves an RMSE of 6.43 and an MAE of 4.98, outperforming the concatenated fusion approach with an RMSE of 6.55 and MAE of 4.99. Notably, the combination of all three NSFs - x-vector, emoHuBERT, and TRILLsson - with FuSeR yields the topmost performance delivering an RMSE of 5.51 and an MAE of 4.48. This highlights the effectiveness of FuSeR in integrating diverse feature sets to enhance model performance.

Table III provides a comparison of FuSeR approach against SOTA on the E-DAIC dataset. The results reveal that FuSeR, using the combination of x-vector, emoHuBERT, and TRILLsson, achieves the lowest RMSE of 5.51 and MAE of 4.48, outperforming previous SOTA work. For instance, the GRU model with DS-VGG and BoAW-e features achieves higher RMSE values of 9.33 and 8.19, respectively, demonstrating that FuSeR not only improves upon current methods but sets a new benchmark in depression detection performance.

In our additional experiments, we evaluated the generalizability of the proposed FuSeR framework on the ANDROID dataset [25], a recent Italian dataset designed for depression detection. This dataset includes tasks such as Interview (I) and Reading (R). Notably, the proposed framework, FuSeR also demonstrates strong performance in a binary depression detection (Yes/No) scenario, highlighting its versatility and robustness across different types of tasks. For all the experiments on ANDROID, we follow the same pre-processing steps, modeling, and training details as followed in E-DAIC.

Table IV presents the results for downstream models trained on individual NSFs for the ANDROID dataset. Among the NSFs evaluated, TRILLsson consistently demonstrates superior performance across all models. Specifically, CNN with TRILLsson achieves the highest Accuracy and F1-score for both the Interview (80.52% and 78.68%) and Reading (75.13% and 73.94%) tasks, indicating that TRILLsson features provide the most robust representation for depression detection. In contrast, x-vector features show relatively lower performance, with CNN achieving 66.32% Accuracy and 64.71% F1-score for the Interview task, and 74.11% Accuracy and 70.12% F1-score for the Reading task.

Table V showcases the performance of various feature fusion techniques on the ANDROID dataset. The proposed FuSeR framework significantly outperforms other fusion methods such as Early, Average, and Concatenation across all feature combinations. For example, with the combination of x-vector and emoHuBERT features, FuSeR achieves an Accuracy of 68.91% and an F1-score of 67.73% for the Interview task, and 75.38% Accuracy and 75.31% F1-score for the Reading task, surpassing the concatenated approach (65.39% and 64.29% for the Interview task, and 75.13% and 72.19% for the Reading task).

When combining x-vector, emoHuBERT, and TRILLsson through FuSeR, we achieve the topmost results, with an Accuracy of 87.93% and an F1-score of 87.84% for the Interview task, and 84.72% Accuracy and 83.35% F1-score for the Reading task. These results underscore FuSeR’s capacity to leverage complementary information from different NSFs effectively and setting a new SOTA on ANDROID dataset. Overall, these additional experiments validate the robustness and adaptability of the FuSeR for improved depression detection.