Conformal Depression Prediction

Over the past decade, depression prediction based on deep learning has attracted significant interest and attention in the field, leading to many meaningful research advancements. Studies using labeled depression data from various modalities, including speech, text, video, and EEG signals, as well as the integration of these modalities, have taken a leading role.

For single-modality depression prediction, He et al. employed a Deep Convolutional Neural Network (DCNN) to extract depression-related features from speech data [21]. These learned features were subsequently integrated with additional hand-crafted features to predict depression scores. Zhao et al. introduced a hybrid network that integrates Self-Attention Networks (SAN) and DCNN to handle both low-level acoustic features and 3D Log-Mel spectrogram data [22]. Following this, feature fusion techniques along with support vector regression are employed for depression prediction. Melo et al. concentrated on efficiently extracting spatiotemporal features from facial videos associated with depression [17, 18]. They introduced a deep Multiscale Spatiotemporal Network (MSN) and a deep Maximization-Differentiation Network (MDN) to predict depression levels. Sharma et al. proposed CNN-LSTM hybrid networks to extract depression features from EEG signals [29]. Seal et al. developed the DeprNet, a DCNN designed for classification of EEG data from depressed and normal subjects [30].

For multimodal depression prediction, Niu et al. introduced a novel Spatio-Temporal Attention (STA) network and a Multimodal Attention Feature Fusion (MAFF) strategy to acquire a multimodal representation of depression cues from facial and speech spectrum [31]. Uddin et al. introduced a novel deep multi-modal framework that effectively integrates facial and verbal cues for automated depression prediction [32]. They extracted features separately from video and speech and employed a Multi-modal Factorized Bilinear pooling (MFB) strategy to efficiently fuse these multi-modal features. Niu et al. proposed a novel multi-layer perceptron (MLP) architecture named DepressionMLP, designed to automatically predict depression levels through facial keypoints and action units (FKRS and AURS) [33].

These studies, either using single-modality data or multimodal data, primarily focus on the prediction accuracy, while overlooking the analysis of predictive uncertainty. Interestingly, works such as those of Melo et al. introduced minimizing expected error to learn distribution of depression levels [34], while Zhou et al. proposed DJ-LDML to learn depression representation using a label distribution and metric learning [20]. These studies implicitly explore uncertainty in depression prediction, yet without providing a clear and intuitive uncertainty quantification. The method closely related to ours is the one proposed by Nouretdinov et al. [35], which leveraged conformal prediction for uncertainty quantification of MRI-based depression classification. Different from the work [35] that dedicated to the classification task, we focus on regression task in the context of deep learning, and we further develop an approximate conditional coverage (ACC) algorithm to overcome the shortcomings of conformal prediction with marginal coverage.

Uncertainty has been widely considered for various tasks in affective computing, such as emotion recognition, facial expression recognition, and apparent personality recognition [36, 37, 38, 39, 40, 41, 42]. In emotion recognition, Harper et al. utilized a Bayesian framework to support emotional valence classification from the perspective of uncertainty [36]. Prabhu et al. proposed BNNs to model the label uncertainty based on subjectivity in emotion recognition [37]. Wu et al. employed deep evidence regression to jointly model aleatoric and epistemic uncertainties, aiming to enhance the performance of emotion attribute estimation [38]. Tellamekala et al. introduced calibrated and ordinal latent distribution to capture modality-wise uncertainty for multimodal fusion in emotion recognition [43]. Lo et al. utilized probabilistic uncertainty learning to extract information from low-resolution data and proposed the emotion wheel to learn label uncertainty, yielding robust facial expression recognition [39]. In facial expression recognition, She et al. adopted a pairwise uncertainty estimation approach, assigning lower confidence scores to more ambiguous samples to address annotation ambiguities [40]. Le et al. proposed a method based on uncertainty-aware label distribution to adaptively construct the training sample distribution [41]. Additionally, Tellamekala et al. utilized a neural latent variable model to model aleatoric and epistemic uncertainty and integrated both to enhance the performance of apparent personality recognition [42].

The uncertainty-inspired algorithms mentioned above may not be suitable for depression prediction, as the available datasets with annotated depression scores are limited. Re-stabilizing a well-performing, uncertainty-inspired depression prediction algorithm is not straightforward. Furthermore, while the above methods may achieve better accuracy performance in their respective tasks by introducing uncertainty-inspired heuristics, they do not establish a rigorously valid uncertainty quantification. In contrast to previous methods, our proposed CDP is a plug-and-play uncertainty quantification method for depression prediction, which does not require model retraining and does not assume the depression data distribution. Thanks to the conformal prediction [26, 44], our CDP can achieve valid confidence intervals with theoretical coverage guarantees for depression predictions.

On the other hand, current conformal prediction methods tend to produce unnecessarily conservative intervals, resulting in the confidence interval to be unnecessarily wide [45]. In depression prediction, overly wide intervals are meaningless. For instance, the prediction with BDI-II (ranging from 0 to 63) has a 100% confidence interval of [0, 63]. Clearly, such a predictive uncertainty lacks practicality. Additionally, marginal coverage guarantees only valid average intervals, which overcovers simple subgroups and undercovers difficult ones [46]. Differently, our proposed conformal prediction method with approximate conditional coverage (CDP-ACC) can produce valid and tighter confidence interval adaptive to the specific depression prediction.

Formally, let us assume that we have the facial depression-associated video data denoted by $X$ as input and the corresponding label denoted by $y$ (the BDI-II score [6], for example). We randomly split the data into disjoint training set $\mathcal{D}_{train}$, calibration set $\mathcal{D}_{cal}=\{(X_{i},y_{i})\}_{i=1}^{N}$, and test set $\mathcal{D}_{test}$. We use $\mathcal{D}_{train}$ to train a DNN model $\hat{f}(\cdot)$ for depression prediction. Here, $\hat{y}\in\mathbb{R}$ denotes the point prediction for $X$ by $\hat{f}(\cdot)$. In the deep learning context, we can formulate the depression prediction as a deep regression task by minimizing the mean squared error (MSE) loss as the learning objective.

where $|\mathcal{D}_{train}|$ denotes the number of data in $\mathcal{D}_{train}$.

In depression prediction based on such regression models, the point prediction $\hat{y}$ is the expectation $\mathbb{E}[y|X]$ derived from estimating the conditional probability distribution $P(y|X)$. However, we do not have access to the oracle of $P(y|X)$. To quantify the uncertainty of depression prediction $\hat{y}$, we expect to construct an interval function $\mathcal{C}_{\alpha}(X)$, which takes depression data $X$ and miscoverage rate $\alpha$ as input and outputs a confidence interval. The interval contains the depression target $y$ with any confidence level $1-\alpha$, as shown in Fig. 1 ($\alpha=0.1$). Clearly, wider interval $|\mathcal{C}_{\alpha}(X)|$ indicates greater uncertainty for the model’s prediction $\hat{y}$.

The straightforward method for uncertainty quantification of depression prediction is to train a DNN to estimate the conditional probability distribution $P(y|X=x)$ of depressed individuals. Given our lack of knowledge regarding the true $P(y|X=x)$, one possible solution is to assume that the depression target $y$ follow a Gaussian distribution $y\sim\mathcal{N}(\hat{y},\sigma^{2})$, and uses the negative log-likelihood (NLL) [47] as the loss function:

The learning objective allows us to train a DNN capable of predicting the distribution $P(y|X=x)$ for any given individual $x$. Furthermore, under the Gaussian assumption, the $\alpha$-th quantile can be calculated by

where $\mathrm{erf}^{-1}$ denotes the inverse Gaussian error function. In this way, we can obtain the confidence interval $[\hat{y}_{\alpha/2},\hat{y}_{1-\alpha/2}]$ that satisfies $P(y\in[\hat{y}_{\alpha/2},\hat{y}_{1-\alpha/2}])\geq 1-\alpha$.

In practice, however, depression datasets (e.g., AVEC 2013 and AVEC 2014 [48, 49]) exhibit high data imbalance, and the assumption of the Gaussian distribution may not be appropriate in this context. We prefer a distribution-free method for uncertainty quantification. Indeed, by introducing the pinball loss (Eq. (4)), we can train a DNN as a conditional quantile estimator [50, 51], allowing us to obtain arbitrary quantile values for $P(y|X=x)$:

where $q\in[0,1]$, $\hat{y}_{q}$ denotes the $q$-th quantile of the prediction, and $y$ is the ground truth. Using the trained regressor, we can use ${\alpha/2}$ and ${1-\alpha/2}$ quantile values of the predictive distribution to derive the confidence interval $[\hat{y}_{\alpha/2},\hat{y}_{1-\alpha/2}]$, thereby achieving a distribution-free uncertainty quantification.

However, due to limited depression data and imbalanced data distributions, using Eq. (2) or Eq. (4) to train depression prediction models is challenging. In practice, it has been observed that those methods suffer from overconfidence, where $P(y\in\mathcal{C}_{\alpha}(X))$ can be significantly lower than $1-\alpha$. In this context, there is an urgent need for a post-processing method that is distribution-free to quantify the uncertainty of depression prediction. In particular, we seek for constructing the confidence interval $\mathcal{C}_{\alpha}(\cdot)$ to guarantee the marginal coverage so as to maintain an average coverage rate across the target, as expressed by

To that end, we introduce conformal prediction (CP [26, 52]) for uncertainty quantification of facial depression prediction. Specifically, we develop conformal depression prediction (CDP) as shown in Fig. 2. As premise of CP is to have a calibration set that is exchangeable with the test sample (This is guaranteed if the i.i.d. assumption holds), we calculate the conformal score $s$ on the calibration set,

which reflects the discrepancy between the depression prediction and the ground truth. Next, we calculate the $q$-th quantile of $s$ and denote it as $\hat{s}$. For any test sample $X_{test}$, we have $P(s_{test}\leq\hat{s})=q$. In this way, we can obtain its prediction intervals $\mathcal{C}_{\alpha}(X_{test})=[\hat{y}_{test}-\hat{s},\hat{y}_{test}+\hat{s}]$, which is valid with

The implementation of CDP are summarized in Algorithm 1. With CDP, we can acquire valid, theoretical coverage guaranteed confidence interval, and the interval width relates to the user-defined confidence levels $1-\alpha$. Note that this CP-based uncertainty quantification achieves marginal coverage for the target depression scores at any confidence levels, as proven in [26, 52].

There is a stronger notion than the marginal coverage, namely conditional coverage, as defined by

For the model prediction $\hat{y}_{test}$ on any test sample $X_{test}$, it aims to provide a confidence interval $C_{\alpha}$ with $1-\alpha$ coverage. In contrast to marginal coverage, conditional coverage is more suitable for uncertainty quantification in individual depression prediction. In practical applications, each individual only cares about whether the prediction is trustworthy (or within which confidence interval the prediction falls). Unfortunately, achieving conditional coverage is practically challenging in a distribution-free setting [44]. Our work draws inspiration from ordinal adaptive prediction sets [53] for classification tasks. The motivation behind our CDP-ACC is to generate adaptive confidence intervals for pre-trained depression predictor, achieving approximate conditional coverage in a model-agnostic and distribution-free manner.

In order to verify the effectiveness of our uncertainty quantification methods in depression prediction, we conduct experiments on two commonly used facial depression datasets, AVEC 2013 [48] and AVEC 2014 [49]. AVEC 2013 is a subset of the AViD-Corpus, consisting of 150 facial videos. The videos range in length from 20 to 50 minutes, with a frame rate of 30 FPS and a resolution of 640 $\times$ 480 pixels. The content of the videos includes facial recordings of subjects during human-computer interaction tasks, such as reading assigned content and improvising on given sentences. We divide the dataset into 70 videos for training, 30 videos for validation (calibration), and 50 videos for testing. The facial depression levels are annotated using BDI-II scores [6], with each video assigned a single value representing the depression level. The BDI-II questionnaire consists of 21 questions, each with multiple-choice answers scored from 0 to 3. The total BDI-II score ranges from 0 to 63: 0-13 indicates minimal depression, 14-19 indicates mild depression, 20-28 indicates moderate depression, and 29-63 indicates severe depression, as shown in Table IV.

AVEC 2014, a subset of AVEC 2013, consists of two tasks: Northwind and Freeform. It comprises 150 Northwind videos and 150 Freeform videos. The original videos of AVEC 2014 are similar to those used in the AVEC 2013 but include five pairs of previously unseen videos, replacing a few videos considered unsuitable. We use 200 videos for training, 50 videos for validation (calibration), and 50 videos for testing.

The videos in the AVEC 2014 dataset are relatively short, with an overlap of 8 frames between adjacent video clips, whereas there is no overlap in the AVEC 2013 dataset. During data processing, landmarks in all video frames are detected using OpenFace [56], and facial region alignment is performed using the eyes, nose, and mouth.

In this section, we provide a comprehensive evaluation of uncertainty quantification methods for depression prediction. Firstly, we show the prediction errors for vanilla regression, NLL and QR, allowing for a comparison of their accuracy performance in depression prediction. Then, we compare the uncertainty quantification performance of different UQ methods in depression prediction. We also perform uncertainty analysis of CDP-ACC for some instances to gain insight into CDP-ACC. Additionally, in the Appendix, we experimentally evaluate the proposed method on several general regression datasets.

The Physicochemical Properties of Protein Tertiary Structure (bio) dataset [60] falls under the subject heading of biology. It consists of data extracted from CASP 5-9 and contains 45,730 decoys ranging in size from 0 to 21 angstroms. This dataset is commonly used for multivariate regression analysis.

The BlogFeedback (blog) dataset [61] belongs to the field of social science. It comprises 60,021 instances, each consisting of features extracted from blog posts and a target variable representing the number of comments the post received within 24 hours. This task is a multivariate regression task commonly used to predict the number of comments a post will receive.

The Facebook Comment Volume Dataset (fb1, fb2) [62] is similar to the blog dataset in that it aims to predict the number of comments on a post. It comprises 40,949 instances, each with 53 features. The dataset includes 5 variants; however, we only utilized the first two variants (fb1 and fb2) in our experiment.

The Medical Expenditure Panel Survey (meps 19 - 21) [64] is a series of large-scale surveys conducted on households, individuals, healthcare providers, and employers across the United States. MEPS serves as the most comprehensive data source on healthcare expenses, utilization, and the extent of medical insurance coverage. In our experiments, we exclusively utilized data from panels 19 to 21.

Our experimental setup follows [45, 52], where we train a conditional quantile estimator (implemented as a three-layer neural network) to handle the regression task described above. We employ the pinball loss (Eq. 4) as our loss function and ensure that the output quantiles are sorted to prevent crossing.

We train the neural network using the Adam optimizer with a learning rate of 0.02. The output quantiles serve as part of the post-processing calibration for conformal prediction. Unlike SCP [26, 44, 27] and CDP-ACC, which do not rely on quantiles, we typically treat the 50% quantile as the output for vanilla regression. We evaluate using Prediction Interval Coverage Probability (PICP) and Mean Prediction Interval Width (MPIW) as metrics, with a miscoverage rate $\alpha$ set to 0.1. For specific definitions of these evaluation metrics, please refer to our main paper.

We present the uncertainty quantification results of several methods, including SCP [26, 44, 27], QR [50], CQR [45], CHR [52], and CDP-ACC, on the aforementioned general datasets. QR and CQR are introduced in the main paper. CHR is a histogram estimation method that calibrates the QR method by constructing nested intervals and integrating them with conformal prediction.

From Table V, it is clear that SCP generally achieves marginal coverage but tends to construct unnecessarily conservative intervals. In contrast, in general regression tasks, QR maintains normal coverage rates and effectively reduces the mean interval width (However, in depression prediction task, QR exhibits a trend of overconfidence, with quantile values overly concentrated around predictions, leading to lower coverage rates.)

CQR and CHR serve as calibration methods for QR, aiming to minimize interval width while maintaining adequate coverage rates. However, CQR is constrained by the simplicity of QR in constructing conformal scores. In contrast, CHR integrates histogram estimation with conformal prediction, enabling more refined optimization of prediction intervals.

For QR, CQR, and CHR, retraining the QR model is crucial. However, vanilla regression models with point prediction can only allow for obtaining prediction intervals through post-processing methods like SCP. Nevertheless, SCP has significant drawbacks such as fixed interval width and overly conservative intervals (unnecessarily wide). CDP-ACC, as an adaptive post-processing method, can be applied to both QR and vanilla regression models. It achieves a finer approximation of conditional distributions through binning, thereby further reducing interval width. From Table V, we observe that CDP-ACC achieves competitive uncertainty quantification results, demonstrating its effectiveness on general regression tasks..