Predicting Parkinson’s Disease with Multimodal Irregularly Collected Longitudinal Smartphone Data

The goal of the multimodal feature extractor is to encode raw activity test results into enriched feature embeddings as the model inputs for end-to-end learning. Considering the sequential data samples conducted in each of the activity tests, e.g. the accelerometer sequence generated when the participant is walking, we decide to adopt the widely adopted Temporal Convolutional Networks (TCN) [20] as the raw feature extractors. In detail, given an observed test sequence $x_{t}^{m}=\{x_{0},...,x_{L_{mt}}\}$ with length $L_{mt}$ from modal $m\in M$ at observation time $t\in T$ , each layer of TCN applies a dilated convolution on the test sequence with a focus on the longitudinal causal relationship:

where $d$ is the dilation rate, and $k$ is the filter size. Then the $n$th element in the $l$th layer is computed with the convolutional kernel $f$ applied on the $n-d*i$ elements in the $l-1$th layer. In this way, we obtain the embeddings $v_{t}=\{v_{t}^{0},...v_{t}^{m}\}$ with $v_{t}^{m}\in\mathbb{R}^{D\times 1}$ for each modality $m$ at observation time $t$.

Traditional ways of handling irregularly sampled data rely on direct aggregations or data imputation strategies. However, these methods are likely to cause information loss at important sequential and continuous signal changes, which are crucial for health condition analysis. Given the extracted and aligned multimodal features at each time point, we propose to model continuous PD symptom changes in the latent space through the Neural Ordinary Differential Equations (ODEs) [16, 17].

Recall that for ODEs, time-series is represented by a latent trajectory determined by the initial state $h_{0}$. Given the observed time points $t_{0},t_{1},...,t_{T}$ and an initialized state $h_{0}$, an ODE solver computes $h_{1},...,h_{T}$ representing the hidden states for each time point. Formally,

where function $f$ produces the gradient $\frac{\partial h(t)}{\partial t}=f(h(t),\theta_{f})$ which is parameterized with a neural network. Each hidden state $h_{t}$ is then obtained by integrating the gradient through time, which is achieved by an ODESolver. To incorporate the observations at each time point $t$ and adjust the latent trajectory accordingly, hidden states are updated by a network, e.g. an RNNCell:

where $h_{t}^{\prime}$ is the hidden state before the update and $u_{t}$ is the observation features at current time $t$.

A simple way to construct input $u_{t}$ is by directly concatenating the embeddings $v_{t}$ extracted from the original observations. However, since participants are free to choose the types of activity tests to perform at home, the observed test results at each time-point are usually incomplete, resulting in some of the modal features being missing. Therefore, we attach each of the modality features with a binary mask $v_{t}^{\prime m}$ with the same feature length, indicating its observation status: $v_{t}^{m}\leftarrow[v_{t}^{m}\cdot v_{t}^{\prime m}]$.

In addition, different modalities may contribute differently to the final PD prediction due to abnormal or noisy test results. Given the hidden state $h_{t}^{\prime}$, a valid modality test should share common representations that consider the similar semantics, e.g. the identity of the same participant. Based on this observation, we propose to integrate an attention mechanism inside the GRU cell, named M-GRU, to further adaptively learn an aggregation function by assigning a weight to each of the modalities:

where

where $w_{m}$, $W_{hm}$, $W_{vm}$ and $b_{m}$ are learnable parameters for computing the transformed representaion $e_{m}$. In this paper, we adopt the GRU unit as the RNN cell for updating the hidden state $h_{t}$. The state-wise input is a concatenation of the original inputs and the attended representations: $u_{t}=[v_{t}\cdot u_{t}^{\prime}]$. Therefore the updating function can be written as follows:

where $W_{z}$, $W_{r}$ and $W_{g}$ are learnable parameters, $\sigma$ is the hyperbolic tangent function.

Different from the observations in controlled environments, e.g. ICUs in hospitals, self-reported test results suffer from poor quality control. Adopting one single state as the user representation for prediction can be easily biased by certain noisy observations. To increase our model’s robustness and extract raw symptom clues from each modality, we adopt a self-attention mechanism [21] on all the encoded modality features at each step $v_{t}$ to form a time-wise global representation:

where

where $v_{t}$ is a concatenation of the extracted modal features $v_{t}^{m}$, while $w$ and $W$ are learnable parameters. We then concatenate the time-wise representation with the last hidden state from the temporal encoder as user representation logits. The final prediction is obtained by applying the sigmoid function on the transformed logits:

We adopt the standard binary cross-entropy loss on the predicted logit $\hat{y}$ and the target label $y$:

We first give a brief review on the mPower dataset [3]¹¹1https://www.synapse.org/#!Synapse:syn4993293/wiki/247859. It contains four types of PD-related activity test results from the participants conducted on their smartphones. In this study, we adopt three of them and leave the more complicated voice signals for future work:

Each of the tests also asks participants to choose their medication points when conducting the test, namely: Immediately before Parkinson medication, Just after Parkinson medication (at your best), I don’t take Parkinson medications, and Another time. For the tapping and walking test, we adopt the accelerometer readings which contain $(x,y,z)$ coordinate sequences in Gs. For the memory test, we use both the tapping response sequences, the corresponding targets and the time spent for the response.

The mPower dataset is collected by the participants outside hospitals with limited quality control. To achieve our goal of multimodal time-series analysis, careful data pre-processing is crucial to remove noisy signals. Due to a large variation on the time when the test is conducted, temporally synchronize test results across different modalities to obtain multimodal observations at unified time points is also needed. We preprocess the collected data as follows:

We compare the proposed method with six baseline models. Three of them are traditional methods while the other three are deep learning based including the state-of-the-art time-series analysis methods RNN+$\Delta t$, GRU-D, and ODE-RNN.

• •

  LR [24]: We leverage a standard logistic regression classifier for binary classification.

• •

  SVM [25]: We adopt a standard Support Vector Machine classifier with the RBF kernel for comparison.

• •

  XGBoost [26]: It stands for Extreme Gradient Boosting which is a tree-based boosting algorithm.

• •

  RNN+$\Delta$t: We concatenates intervals $\Delta$t to the input features and feed them into a standard RNN model [27]. The last hidden state is used for the final prediction.

• •

  GRU-D [13]: The GRU-D model is also designed for modeling trajectory changes with a hidden state exponentially decay through time. In addition, it concatenates observational masks and time intervals between the observations as additional clue into the inputs.

• •

  ODE-RNN [17]: The ODE-RNN model focuses on the continuous latent space trajectory modeling which captures inter-observation changes with ODE and at-observation hidden state updates with a GRUCell.

Baseline methods LR and SVM are adopted from the Scikit-Learn library. The XGBoost algorithm is adopted from its publicly available python version. Since LR, SVM and XGBoost algorithms are not designed for handling irregular time-series inputs, we take the average of the features from all the steps as the global representation and feed them into these classifiers. For each step, we concatenate three modalities as a combined representation. The remaining methods use the same inputs and the same settings for the feature extractor and final prediction network. During training, the Adam optimizer is used with a learning rate initialized as 0.01 and decay by 0.96 for each epoch. All experiments are conducted with 5-fold cross-validation. Area Under the Curve (AUC), Area Under the Precision-Recall Curve (AUPR) and F1 scores are used as the evaluation metrics. We report the average and standard deviation values obtained from cross-validation. The proposed model is implemented in PyTorch and experimented on a single NVIDIA Geforce GTX 1080Ti GPU.