Interpreting Depression from Question-wise Long-term Video Recording of SDS Evaluation

We collect the Self-Rating Depression Scale (SDS) questionnaires and the corresponding face video from 200 subjects. The study protocol was approved by the Ethics Committee of the Affiliated Hospital of Guangdong Medical University (No. PJ2021-026).

Each participant is instructed to sit in a quiet consulting room along and fill the self-report questionnaire following the instruction in the software interface to avoid being affected by the others. Moreover, a Huawei Software-Defined Camera (SDC) is hidden behind a one-way mirror, and the participants do not become aware of the camera in the evaluation. SDC adopts an open-ended software architecture that can flexibly integrate with the code of specific application²²2https://e.huawei.com/en/products/intelligent-vision/cameras/software-defined-camera. In addition, the data is uploaded to the back-end server for processing. To connect each question with the video, the camera is synchronized with the questionnaire software to record the video start from the question showing and end on the score is chosen. The illustration of our data collection is shown in Fig. 1. In Fig. 2, we provide some frames of a subject for answering a question.

There are 20 questions for SDS evaluation, and they usually take about 10 minutes to complete[7]. Each participant takes a different time to finish different questions. In our collected dataset, the time of each question varies from 2s to 21s. The 25 fps videos with the resolution of 3840$\times$2160 are collected for each question. Since the region of interest is the face of participants, we use the face detector [52] to crop the face region. To incorporate the hand action of head-scratching [53], chin-touching [54], etc., we extend the face box by 100% and resize the extended region of interests (ROI) in each frame to 110$\times$110. Moreover, the image is followed by gray processing to reduce the size of the input. The pre-processing flow chart is given in Fig. 3.

After the self-report SDS evaluation, a more comprehensive clinical interview [3], including the clinician-administered Hamilton Depression Rating Scale (HDRS) assessment [11], SCL-90-R Symptom Checklist [55], and Self Rating Anxiety Scale (SAS)[56], is taken to help the clinician for confirming a diagnosis of depression. We use the final diagnosis result as our ground truth label.

Considering the self-administrated SDS can be uncontrollable, the result of SDS evaluation can be different from the final diagnosis [9]. In Tab. I, we provide the detailed statistics of the SDS and final results of our 200 subjects. We note that there are only two classes, i.e., normal and depression, in our dataset. We can see that there are about 10% of subjects have different results. Specifically, the SDS evaluation of 10 subjects is diagnosed as depression, while the other 10 subjects with the high SDS score are diagnosed as normal with the subsequent clinical interview.

The long-term video recording of SDS evaluation inherits rich emotional information, while it also poses several challenges for processing. First, the long video can be redundant, and only very few (i.e., sparse) frames may indicate the useful cues for depression detection. Second, the length of the video can be varied across different subjects and questions.

Considering that the human expression usually takes 200ms to 500ms, it can be reasonable to segment the video into several fixed-length short clips and explore the local temporal patterns within it. We empirically set each video clip to 10 successive frames in our task. With the fixed length, the 3D CNN can be an efficient module for fast processing [42].

To avoid splitting an expression, we set an overlap ratio to 0.5, that the first clip of the first question $\{I^{1}_{1-i}\}_{i=1}^{10}$ is from 1 to 10 frames, and the second clip $\{I^{1}_{2-i}\}_{i=1}^{10}$ is from 6 to 15 frames, respectively. We use the superscript to denote the question from 1 to 20, and the subscript denotes the images in each clip. Therefore, for a $N$-frame video, there can be $M=2\times(N/10)-1$ clips. For a 16 seconds video with 25fps in our dataset, we have 79 clips, and only very few of them are not neutral expressions and do not contribute to complementary information. We extract the 128-dimensional feature of each clip and denote the clip-wise representation of the first question as $\{f^{1}_{m}\}_{m=1}^{M}$, where $m\in\{1,\cdots,M\}$ index the clips within a question. We note that $M$ can be different for different questions, and we omit the question index for simple notations.

After the clip-wise representations are extracted, the global attention is applied on top of them to extract the 128-dimensional question-wise features $\{a^{i}\}_{i=1}^{20}$ for 20 questions. To adaptively learn the significance of each clip within a question, we resort to a redundancy-aware self-attention scheme.

Then, the question-wise features $\{a^{i}\}_{i=1}^{20}$ are concatenated with the corresponding tabular questionnaire results and the answering time. We note that the SDS has 4 scales, and we use four-dimensional one-hot vector to encode the choice of each problem. The answering time is also concatenated as a one-dimensional scalar. The concatenated feature of each question can be a 133 dimension vector, and all of the 20 questions are concatenated together to form a 2660-dimensional questionnaire and video fused feature. We use the fully connected layers with the sigmoid output unit for binary classification, i.e., normal or depression.

We note that all of the 3D CNN and self-attention modules are shared for all of the clips and questions. In the following subsections, we provide the detailed framework of our hierarchically constructed 3D CNN, redundancy-aware attention, and question-level fusion modules.

3D CNN has demonstrated its effectiveness of fast temporal representation extraction for relatively short fixed-length video [18]. We apply the standard 3D convolution operation to model the relationships between the successive frames. We illustrate the basic framework of 3D CNN in Fig. 5. After a few 3D Convolutional and maxpooling layers, we get a 256-dimensional feature vector, which is sent to a fully connected layer and result in a 128-dimensional clip-wise representation $f^{i}_{m}$, where $i$ and $m$ index the 20 questions and $M$ clips within this question, respectively.

Considering the height and width dimensions of the video clips (e.g., 110) is usually much larger than the frame-wise dimension (e.g., 10), the first three maxpooling layers only half the height and width dimensions and denote as (2$\times$2$\times$1)-(1$\times$1$\times$1). In the 4-th maxpooling layer, we half all of the three dimensions and denote as (2$\times$2$\times$2)-(1$\times$1$\times$1). The detailed network structure is given in Tab. II.

To explore the correlations between each clip, we resort to the affinity of clip-wise feature vectors. We use $i$ to index the $M$ clips within a question, and the $i$-th vector is regarded as a probe vector. $j$ index the other $M-1$ clips other than the $i$-th clip. We note that different questions can have a different number of fix-length clips, while the 3D CNN is not applicable [57, 42].

In our redundancy-aware self-attention module, we configure several self-attention blocks, which are indexed by $l\in\{1,2\cdots,L\}$. $L$ is the total number of stacked sub-self attention blocks. In each self-attention block, we first traverse all of the clips to set the current clip as the probe. Then, we traverse the M-1 clips other than the probe to explore their correlations with the current probe clip.

Specifically, our self-attention block can be formulated as

where ${\Omega^{(}l)}\in\mathbb{R}^{1\times 1\times 128}$ is the weight vector to be learned and ${f}_{i}^{q(0)}={f}_{i}^{q}$. The response is normalized by ${C_{i}}$. The pairwise affinity $\omega(\cdot,\cdot)$ is an scalar.

The operation of $\omega$ in Eq. (III-D) has many possible function candidates [19, 51]. We simply choose the embedded Gaussian given by

where $\psi({f_{i}^{q(0)})={\Psi}f_{i}^{q(0)}}$ and $\phi(f_{j}^{q(0)})={\Phi}f_{j}^{q(0)}$ are two embedding functions, and $\Psi$, $\Phi\in\mathbb{R}^{128\times 128}$ are the corresponding learnable mapping matrix [19].

The residual term, i.e., $f_{j}^{q(l-1)}-f_{i}^{q(l-1)}$, is the difference of the neighboring feature ($i.e.,f_{j}^{q(l-1)}$) and the current probe feature $f_{i}^{q(l-1)}$. If $f_{j}^{q(l-1)}$ incorporates complementary information or more significant cues compared to the current probe feature $f_{i}^{q(l-1)}$, then our redundency-aware attention scheme will eliminate the information from the inferior $f_{i}^{q(l-1)}$ and emphasis the more discriminative $f_{j}^{q(l-1)}$. Compared to the original non-local network which uses only $f_{j}^{q(l-1)}$ [19], our formulation can be more similar to the diffusion maps [58], graph Laplacian [59] and non-local image processing [60]. All of them are non-local analogues [61] of local diffusions, which are expected to be more stable than its original non-local counterpart [19] due to the nature of its inherit Hilbert-Schmidt operator [61]. We not that the the current probe feature will be added back in the last as the residual neural network. Therefore, the previous steps in the self-attention block is adjusting the information of each clip that need to be emphasised and translate to the later blocks.

Since the pair-wise residual term takes all possible clips into consideration and not following the sequential modeling, it does not suffer from long-term forgetting. Therefore, it can be an ideal choice for the attention modeling of many clips. In addition, the weighted average operation is able to take any number of inputs, which is fit for the different clip numbers in different questions. We note that the input and output of a self-attention block have the same size. Specifically, the $M$ inputs with the size of $1\times 1\times 128$ will be processed to $M$ outputs with the size of $1\times 1\times 128$.

Permutation-invariant is a special property of a self-attention scheme [39]. Since we use the sum operation in Eq. 1 to fuse the pair-wise residual terms. In the previous self-attention-based video analysis works, each frame is regarded as independent of each other and discards the sequential patterns [19]. Our video clips are inherently sequential data, and have an overlap with the neighboring clips. For exploiting the temporal patterns, the Gaussian kernel is proposed for sequential neighboring distance measure

where $m_{i}$, $m_{j}\in\mathbb{R}$ represent the position of $i^{th}$ and $j^{th}$ feature vectors in the video of a question, respectively. $\sigma$ is a hyperparameter to control the shape of Gaussian Kernel.

After several concatenated self-attention blocks, the global pooling [19] is added on these $M$ feature maps for element-wise averaging. The final output is a question-wise video feature $a^{q}\in\mathbb{R}^{1\times 1\times 128}$.

The SDS questionnaire score of each question is denoted as $s^{q}\in\mathbb{R}^{4}$. Moreover, we empirically found that the answering time of each question can also be helpful for the diagnosis. Therefore, we also concatenate the video length of each question $t^{q}\in\mathbb{R}$. For a video that takes 3 seconds, we set its $t^{q}$ to 3. The too short or too long answering maybe unreliable [17].

We concatenate $a^{q}$, $s^{q}$ and $t^{q}$ to form a 133 dimension feature vector for each question. Then, the 20 question-wise video and questionnaire features are concatenated to a 2660 dimensional feature for a final depression diagnosis. The fully connected layers are adopted, and the detailed network structure is given in Tab. III. The widely used Rectified Linear Unit (ReLU) [62] is used as the non-linear mapping function between each fully connected layer.

We use the sigmoid output unit $p=\frac{1}{1+e^{out}}\in(0,1)$ for binary classification, where $out$ is the network prediction scalar in the last layer and will be normalized to a probability value, i.e., the likelihood of this subject is depression patient. We note that we indicate the normal subject with 0 (i.e., $y=0$) and the depression subject with 1 (i.e., $y=1$) according to the final clinician interview. To automatically train our model with backpropagation, we use the binary cross-entropy loss as the optimization objective.

which has the zero loss if $p$ matches its corresponding $y$. In addition, we simply set the threshold of binary prediction to $p=0.5$ for testing.

All the experiments were implemented using the widely adopted deep learning library Pytorch [63] on our server with an NVIDIA V100 GPU, Xeon E5 v4 CPU with 128GB memory. Our model is trained with the Adam [64] optimizer with the hyper-parameters of $\beta_{1}$=0.9 and $\beta_{2}$=0.999. We used a batch size of 2 for our dataset. The networks of our framework and the compared methods are trained for 200 epochs for a fair comparison. We report the results of five random initialization, and provide the standard deviation ($\pm sd$) along with the average performance. We note that the 3D CNN and the redundancy-aware self-attention modules are shared for all of the questions, which can be processed parallelly. We empirically set $L=5$. The training takes about 8 hours, while the average inference time of a subject only takes 1.3s. The threshold of binary classification testing is set to 0.5.

We adopt the five-fold cross-validation for the 200 subjects in our dataset. Specifically, we split the dataset into five subsets, and each has 40 subjects. We note that there are no overlap w.r.t. subjects between two folds. Then, we sequentially select a fold as our testing set (i.e., 40 participants), while the remaining four folds (i.e., 160 participants) are used for training.

For performance evaluation, we use the widely accepted binary classification metrics of accuracy, sensitivity (i.e., recall), specificity. More formally:

where TP, TN, FP, and FN indicate true positive, true negative, false positive, and false negative, respectively. We note that the positive and negative corresponding to the depression and normal, respectively.

In addition, by varying a threshold, the receiver-operating characteristic (ROC) curve plots the true positive rate (TPR) against the false positive rate (FPR). It demonstrates the diagnostic performance of a binary classification algorithm. The larger area under the curve (AUC), the better performance. We note that the TPR$=\frac{TN}{TN+FP}$ and FPR$=\frac{FP}{FP+TN}$.

With our SDS and video dataset, there can be three choices of the modality, i.e., SDS only, video only, and both of them. With only the SDS evaluation result, we can simply add the score of 20 questions and using the threshold of 50 for normal and depression classification. According to the statistics in Tab. I, we can see that the SDS results can be different from the clinician diagnosis. We also tried only using the video modality for classification, which did not concatenate $s^{q}$ for the question-level fusion.

It is clear that using both SDS and video can outperform SDS only by a large margin, which evidenced the effectiveness of the additional video modality. The complementary information in the corresponding video is helpful to detect depression more accurately. It is also appealingly that even we only use the video, we are also able to predict the depression with an accuracy of 0.69, which is higher than the chance probability of 0.5.

To demonstrate the effectiveness of our model, we also applied two baseline methods, i.e., recurrent neural networks (RNN) [40], and non-local networks [19], for comparison. We note that this is also the first attempt to apply RNN and non-local attention for SDS and video analysis.

RNN is a typical choice for temporal modeling [65]. Specifically, we use the bi-directional LSTM [41] for our video feature extraction. Moreover, the non-local scheme [19] is recently proposed to address the long-term forgetting of RNN [66]. We use RNN or non-local to replace the 3D CNN and redundancy-aware self-attention in our framework to extract the 128-dimensional question-wise feature representation. The quantitative evaluation results are shown in Tab. IV. In addition, the accuracy of different epochs is plotted in Fig. 8. Our proposed framework achieves significantly better performance than the RNN and non-local counterparts.

We also provide the systematical ablation study for our framework modules.

$\bullet$ Our:color indicates using RGB frames as input, which without the gray-scale pre-processing. We note that we can simply modify the 3D CNN for multi-channel input, while the computation cost can be significantly increased.

$\bullet$ Our:non-local indicates using the conventional non-local [19] as the alternative of our redundancy-aware self-attention module. We note that the original non-local [19] is also first introduced to depression detection in this paper, and is regarded as a baseline.

$\bullet$ Our:w/o time denotes the $t^{q}$ is not concatenated for the question-level fusion.

$\bullet$ Our:$f_{j}^{l-1}$ refers to using ${\footnotesize f_{j}^{l-1}}$ instead of the difference term $f_{j}^{q(l-1)}-f_{i}^{q(l-1)}$ in Eq. (III-D). It does not explicitly consider the redundancy and lead to lower accuracy.

$\bullet$ Our:$(\Psi\Phi)^{L}$ indicates using the embedded Gaussian pair-wise affinity for every block, which has similar performance but usually doubles the training time.

$\bullet$ Our:w/o$\Delta$ indicates excise $\Delta_{i,j}$ in Eq. (III-D) and does not taking the temporal sequence into consideration.

$\bullet$ Our+gender indicates that we add the 1/0 label of male/female along with the time.

$\bullet$ Our:MLP indicates that only using MLP to fuse the score and SDS time.

$\bullet$ Our:SLF indicates that using score-level fusion instead of feature level fusion.

The results are provided in Tab. V. The relatively inferior performance of the compared settings demonstrates the effectiveness of our choices. By adding the gender label, we do not achieve improvements w.r.t. the accuracy and AUC metrics. Our proposed method is able to explore the video information and achieve better performance than using MLP to fuse the score and SDS time. In addition, our feature-level fusion can outperforms the score-level fusion significantly.

There are several hyperparameters in our framework. We provide a sensitivity analysis of these settings in this subsection and provide the analysis results in Fig. 9.

Specifically, we tested using different output dimensions of 3D CNN to explore the balance of the representative and computational cost. In Fig. 9, we can see that the output dimension of 3D CNN can be stable between 128 to 256 dimensions. The longer output feature can introduce significant additional costs for the subsequent redundancy-aware self-attention scheme. Moreover, since the subsequent redundancy-aware self-attention scheme does not change the length of the feature, the fully connected layers can be hard to process the longer inputs without enlarging its network structure.

The length of each clip can be related to the developing time of human expression and action in this task. The performance is not sensitive for a large range, e.g., 8 to 13 frames in a clip. The too-short clip may not able to incorporate an expression or action, while the longer clip can be hard to be effectively processed to extract useful information.

In addition, we can use different $L$ to configure the number of redundancy-aware self-attention blocks. With five self-attention blocks, it is able to achieve the best performance in our task.

In Table VI, fix the 40 subjects in each cross-validation round, and reduce the training sample in each round to 40, 80, and 120 subjects. The performance can be better with more training data. We note that the difference between using 120 subjects or 160 subjects can be similar.

In Table VII, we compared the performance of using different head box sizes. The performance is not sensitive to the size within a relatively large range, while the 200% can be a good balance of the efficiency and performance.

In Table VIII, we investigated the performance of using different fps of the video. We can see a significant performance drop w.r.t. both accuracy and AUC for the lower fps. Therefore, we chose the highest fps in our dataset. It can be promising to apply a higher frequency camera to capture the micro-expression information, while it can be costly in computation and memory.

Depression has been affecting more than 300 million people worldwide, while the early diagnosis can be immensely helpful for the treatment. The popularity of depression is even more significant in the COVID-19 season [4], which has long-term quarantine rules. A recent COVID-19 mental health survey [5] indicates that 23% of adults in Ireland reported suffering from depression³³3https://www.maynoothuniversity.ie/newsevents/covid19mentalhealthsurvey/maynoothuniversityandtrinitycollegefindshighratesanxiety. However, the clinician interview can be prohibited or difficult considering the restrictions of avoiding infectious diseases, e.g., COVID-19. This further requires efficient self-administrative screening for depression detection.

The proposed framework has demonstrated good prediction accuracy for normal and depression, and has the potential for clinical practice in the future, especially for self-screening. With the Software-Defined Camera (SDC), we are able to transfer the questionnaire and its video to the back-end server for processing. Moreover, a similar protocol can be potentially applied to the smartphone APPs, which can easily take the face video of the user with the front camera. We note that the captured view can be different from our collected data and may result in the domain shift of the appearance. A possible solution to avoid the largen scale labeling of the mobile captured video is using the unsupervised domain adaptation to transfer the knowledge from our dataset to the unlabeled mobile dataset [67, 68, 69]. In addition, it is promising to apply a face pose invariant or robust feature extractor as [35, 70, 71, 72]. Therefore, the subsequent video-level aggregation modules can be shared across the datasets with different face poses. Actually, our pre-processing only crops a small area of the head, which is robust to the background changes. We are able to adjust the threshold for different applications with different sensitivity to the misclassification. Positive patients will then be referred to specialized clinics for a more comprehensive diagnosis.

The swift, automated deep learning system will partially substitute and support primary doctors’ long-term clinical training, improving the primary diagnosis accuracy of depression in developing countries and laying the groundwork for early diagnosis and care of depression patients.

Our system targets to explore the SDS evaluation and its video, while the clinician interview usually involves the round-based conversation. The spontaneous reaction and the speech (including text and phoneme) can provide more informative features. The interactive multi-round dialog system can be a promising direction.

In addition, we only collect the subject in China, especially in the Guangdong province, the population shift may affect the performance of our system. we will continuously collect more samples from different areas in the following study to achieve better performance.

Moreover, anxiety is also closely related to depression, while can have different treatment. In future work, we are also planning to incorporate anxiety into our diagnostic system.