Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networks

A database recently collected for a collaborative observational study conducted by the University of Maryland School of Medicine and the University of Maryland College Park has been used for this study (Kelly et al., 2020). The database contains video and audio data of free response assessments administered in an interview format. Data for this study was collected from 23 schizophrenic (SZ) patients and 20 healthy controls (HC). All of the schizophrenic patients were clinically diagnosed. Every subject participated in four interview sessions over a period of six weeks. Each interview session is 10-45 minutes long and every subject is assessed using standard depression severity measures and global psychopathology measures by a clinician and themselves. For this study, we used the clinician assessments based on the 18-item Brief Psychiatric Rating Scale (BPRS) (Hunter and Murphy, 2011), where we selected subjects based on the total BPRS score, and the subscores for psychosis (BPRS item11, item12, item4, item15) and activation (BPRS item6, item7, item17), and the Hamilton Rating Scale for Depression (HAMD) (Gonzalez et al., 2013).

Table 1 presents the information on the dataset used for the study. The 7 schizophrenic subjects (4 Males, 3 Females) are selected such that they are markedly ill (BPRS total $\geq$ 45), have higher sub scores for psychosis and activation, but are not depressed or only mildly depressed (HAMD between 0 and 14). The 11 healthy controls (5 Males, 6 Females) are chosen such that they are not depressed (HAMD $<$ 7) or schizophrenic (BPRS $<$ 32).

The audio data were first diarized using transcripts which include the speaker ID and time stamps to separate out the utterances which correspond to the subject from the interviewer. The utterances which are longer than 40 seconds were then segmented into 40 second chunks. If the remaining amount was less than 5 seconds (the minimum length accepted), then it was added back to the last segment. Thus, for all the classification experiments, we used segments with a minimum length of 5 seconds and a maximum length of 45 seconds.

The video-based Facial Action Units (FAUs) provide a formalized method for identifying changes in facial expressions. We used the Openface 2.0: Facial Behaviour Analysis toolkit (Baltrusaitis et al., 2018) to extract seventeen FAUs from the recorded videos of the subjects during the interviews. The FAU features were sampled at a rate of 28 frames per second. We only analyzed those portions of the video when the subject was talking. For parallel delay CNN model in section 3.1 and FVTC model in section 3.2, we choose only 10 FAUs (FAU 6,7,9,10,12, 14,15,17,20 and 23 from FACS coding system (Prince et al., 2015)) which are near the mouth area that can capture coordination of lip and near lip movements during the voice activity. One other reason to choose 10 FAUs is to handle the high dimensionality of the TDEC correlations structure which poses computational limitation when training multi-modal networks in section 4.3.

We used a speech inversion (SI) system (Sivaraman et al., 2016; Sivaraman, 2017) that maps the acoustic signal into 6 vocal tract variables (TVs). The 6 TVs are namely Lip Aperture (LA), Lip Protrusion (LP), Tongue Body Constriction Location (TBCL), Tongue Body Constriction Degree (TBCD), Tongue Tip Constriction Location (TTCL) and, Tongue Tip Constriction Degree (TTCD).

Seneviratne et al. (Seneviratne et al., 2020) in a recent study showed that incorporating glottal TVs generated by periodicity and aperidocity measures (by Aperiodicity, Periodicity and Pitch (APP) detector (Deshmukh et al., 2005)) improved the results of depression detection. Thus, in this study we use 6 TVs generated from the SI along with the 2 glottal TVs as the key audio features for the classification models.

Previous studies in depression prediction using speech (Ray et al., 2019; Ringeval et al., 2019) have shown the superiority of MFCCs over other audio based features like extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) (Eyben et al., 2016) and DEEP SPECTRUM features (Amiriparian et al., 2017). Huang et al. (Huang et al., 2020) showed with their depression classification study that coordination features computed from MFCCs perform better with respect to formants and eGeMAPS features. So to compare how robust and effective the TVs are for detecting schizophrenia, we chose MFCCs as the baseline audio features for our study. We extracted 13 MFCCs from the librosa python library using an analysis window of 20 ms with a 10 ms frame shift. Only 12 MFFCs were used for analysis by discarding the 1st coefficient.

Coordination among 10 FAUs, 6 TVs and the 12 MFCCs were estimated using the correlation structure features. The features are estimated by computing a channel delay correlation matrix using time delay embedding at two delay scales (Espy-Wilson et al., 2019; Williamson et al., 2019, 2013). The computed correlation matrix can be considered as a compact representation which provides rich information on the underlying mechanisms in articulatory coordination. Each correlation matrix $R_{i}$ has a dimensionality (MN x MN) where M = 10, 8 and 12 for FAUs, TVs and MFCCs respectively. N corresponds to the number of time delays per channel which is 15 for all the considered feature types.

From the correlation matrix $R_{i}$ calculated for each sample $i$, the eigenspectrum is computed. The eigenspectrum generated for FAUs is a 150-dimensional vector which is rank ordered (in descending order of magnitude of eigenvalues) from index j=1,..,150. The eigenspectra generated from TVs and MFCCs are 120-dimensional and 180-dimensional, respectively.

Figure 1 shows the averaged eigenspectra (on left) computed for FAUs, TVs and MFCCs. The difference plots in Figure 1 (in right) are calculated by taking the difference between averaged eigenspectra curve for speech from schizophrenic subjects with respect to that of healthy controls. These eigenspectra and difference plots help us to understand the coordination complexity of the speech and facial gestures. The magnitude of each eigenvalue is proportional to the amount of correlation in the direction of their associated eigenvectors. The difference plots show that schizophrenic speech has smaller low-rank eigenvalues relative to the healthy controls, and the trend is reversed towards the high-rank eigenvalues. Therefore, schizophrenic speech needs a larger number of independent dimensions implying a more complex articulatory coordination than speech from healthy controls (Williamson et al., 2013, 2012). The same argument is true for facial gestures.

Huang et al. (Huang et al., 2020) in a recent study with MDD introduces a new channel delay correlation method inspired by TDEC, which uses a different correlation structure with correlations starting from 0 to a delay of ’D’ frames (a design choice). The delayed autocorrelations and cross-correlations across channels are stacked to form the FVTC correlation structure. FVTC includes every correlation within the considered D frames and also avoids the repetitive use of same correlations as in the TDEC correlation matrix.

We developed a CNN architecture which takes in multiple time-delay embedded correlation matrices with 2 delay scales in parallel as inputs for two 2D-CNN layers. The output from the 2 CNN layers are then concatenated and passed through another 2D-CNN layer. Batch normalization, max-pooling and dropout were applied afterwards. The flattened output is then fed to a fully connected layer with 64 neurons. 16 filters with kernel size (3,3) was used for every 2D CNN layer and every CNN layer has ReLU activation. We trained individual models for FAUs, TVs and MFCCs where 3 and 7 sample delay scales were used for FAUs and 7 and 15 sample delay scales were used for TVs and MFCCs.

We designed a CNN model inspired by the one in (Huang et al., 2020) which takes the FVTC correlation matrix computed in section 3.2 as the input. To reduce the number of trainable parameters in the original model (Huang et al., 2020), we reduced the size of the two fully connected layers to 64 and 8. We used the same dilation rates 1,3,7,15 as in the original model. We chose 45 as the ’D’ parameter for FAUs and 50 for TVs and MFCCs. The ’D’ values were chosen from the set of (45,50,55) after doing a grid search on individual feature based systems.

One of the key contributions of the paper comes with the mutimodal networks developed to fuse the audio and video based coordination features calculated from the TDEC and FVTC methods. 3 types of fusion models were designed namely the (Parallel delay TDEC-FVTC CNN), (FVTC-FVTC CNN) and (Parallel delay TDEC- Parallel delay TDEC CNN). The same model architectures developed in section 4.1 and section 4.2 were used and they are fused after passing through all the 2D-CNN layers by concatenating the flattened outputs from each model. The concatenated output then passes through 2 fully connected layers which have 64 and 8 neurons, respectively. That output is then fed to a softmax output layer to generate the final class probabilities (for schizophrenic and healthy classes).

To choose the learning rate and batch size for the individual and multimodal systems, we did a grid search from sets of (1e-4, 3e-4, 1e-5, 3e-5) and (32,64,128) for learning rates and batch sizes, respectively. 1e-4 for learning rate and 64 for batch size gave the best metrics in classification for the best performing model in Table 3. Every model was trained in leave one subject out cross validation fashion where both accuracy and F1 scores are calculated across 18 folds. This ensured that every model is always trained with around 2000 sample points (17 subject’s data) and then tested on the excluded subject (test fold). To come up with the subject level prediction label from the multiple segment level predictions, we use the best 25% of the total segments in the test fold which are predicted with the highest class probabilities (even if the model predicts the wrong class). Every model is trained for 200 epochs with early stopping based on validation loss with patience of 15 epochs. Figure 2 shows the architecture for the best performing multimodal system from Table 3.