Sequence-based Dynamic Handwriting Analysis for Parkinson’s Disease Detection with One-dimensional Convolutions and BiGRUs

The “Parkinson’s disease handwriting database” (PaHaW) collects handwriting data of 37 PD patients and 38 age and gender-matched healthy control (HC) subjects [17]. Participants were enrolled at the First Department of Neurology, Masaryk University, and the St. Anne’s University Hospital, Brno, Czech Republic. All participants were right-handed, had completed at least ten years of education, and reported Czech as their native language. No significant between-group difference regarding age or gender was found. None of the subjects had a history or presence of any psychiatric symptom or disease affecting the central nervous system, with the exception of Parkinsonism in the PD group. Patients were only examined in their ON-state while taking dopaminergic medication, and, prior to acquisition, they were evaluated by a qualified neurologist. Additionally, the HC group underwent a thorough examination to ensure that no movement disorder or injury could have significantly affected handwriting.

All participants were asked to complete eight handwriting tasks following a pre-filled template:

1. 1.

   Drawing an Archimedes spiral;

2. 2.

   Writing in cursive the letter l;

3. 3.

   The bigram le;

4. 4.

   The trigram les;

5. 5.

   Writing in cursive the word lektorka (“female teacher” in Czech);

6. 6.

   porovnat (“to compare”);

7. 7.

   nepopadnout (“to not catch”);

8. 8.

   Writing in cursive the sentence Tramvaj dnes ǔz nepojede (“The tram won’t go today”).

Since not all participants completed each task, we considered only those subjects who completed each of the eight tasks, i.e. 36 PD and 36 HC.

The handwriting signals were recorded using a Wacom Intuos digitizing tablet, overlaid with a blank sheet of paper. Like many other professional tablets, the raw data acquired are the $x$- and $y$-coordinates of the pen tip, the corresponding time stamps, measures of pen inclination, i.e. tilt-$x$ and tilt-$y$, and pen pressure. The button status is also available, which is a binary variable with value 0 for pen-ups (“in-air movement”) and 1 for pen-downs (“on-surface movement”). The sampling rate was 200 samples per second. Few sample images of healthy and Parkinsonian writing are depicted in Fig. 1.

The NewHandPD database [36] is an extension of the previous HandPD corpus [35]. The first database consisted of images from two drawing tasks, i.e. the typical spiral cognitive test and a modified spiral (“meander”) test performed by healthy individuals and people with Parkinson’s disease. However, the new corpus, NewHandPD, contains both offline images and online signals (time-based sequences) of the two groups. The handwriting signals were acquired through a technology other than a tablet, i.e. an electronic smart pen (BiSP).

Specifically, NewHandPD contains images and dynamic data from 31 patients and 35 healthy people. The gender of the participants was fairly balanced (39 males and 29 female), while most of them were right-handed writers (59 of 66 participants). They were asked to complete a handwriting-based test consisting of the following 12 exams:

1. 1.

   Four tasks related to spirals;

2. 2.

   Four tasks related to meanders;

3. 3.

   Two circled movements (one in-air and another on-surface);

4. 4.

   Two diadochokinesis tests (one with the left hand and the other with the right).

The electronic smart pen recorded the following temporal data in its six channels for each exam:

1. 1.

   Microphone;

2. 2.

   Finger grip;

3. 3.

   Axial pressure of ink refill;

4. 4.

   Tilt and acceleration in $x$ direction;

5. 5.

   Tilt and acceleration in $y$ direction;

6. 6.

   Tilt and acceleration in $z$ direction.

It is assumed that raw time-series data sampled from a conventional digitizing tablet are available: pen position, time stamp, pen pressure, pen inclination, and button status. Kinematic and pressure features can be derived from these raw measures. Kinematic features include the tangential, horizontal and vertical displacement, velocity, acceleration, and jerk. Displacement is the straight-line distance between two consecutive sampled points:

These features are suitable for our classification problem, as several studies, e.g. [9, 43], reported alterations of Parkinsonian handwriting in terms of writing time, writing size, applied pressure, and velocity fluctuations. Note that we do not consider other commonly used spatio-temporal variables, such as stroke size and duration, overall time, etc., as they are expressed as a single-valued feature rather than a time-dependent vector feature [17].

Each handwriting sample $S_{n}$ can therefore be represented as a multidimensional vector of $m$ dynamic features, where each feature $X_{i}$ consists of a sequence of $T$ time-steps:

The time-dependent sequences are fed into one-dimensional (1D) convolutional layers with stride greater than 1. The advantage of employing 1D convolution is two-fold. First, these layers sub-sample the input sequences, thereby reducing the overall training cost of the RNN model. Second, 1D convolutions can extract local temporal information from the input sequences, thus performing a pre-training step towards learning meaningful temporal dependencies. Combining 1D convolutions and RNNs is beneficial, especially when dealing with very long sequences that would hardly be processed with an RNN. In our case, we can have a few thousands time-steps for a sequence. The effect of the convolutional layers is to turn the long input sequence into much shorter (down-sampled) pieces of higher-level, locally invariant features. The sequence of extracted features then represents the input for the RNN component of the network.

Several filters of varying sizes are used in each layer to extract information across multiple time-scales. In particular, we employed two convolutional layers in cascade. The first applies $8$ filters, with a kernel of size $5$ and stride $5$. The following layer involves a higher number of $16$ filters with a reduced kernel of size $3$ and stride $3$. A commonly used ReLU nonlinearity follows both layers. It is common practice to augment the number of filters in the following layers as the low-level features of the previous one can be combined in several ways to obtain higher-level representations.

Recently, deep learning models such as recurrent neural networks have gained popularity in sequential data analysis [47]. Unlike a conventional feed-forward neural network, an RNN has a recurrent hidden state $h_{i}^{t}$, whose activation at a given time $t$ depends on the previous state at time $t-1$. This is shown in the following equation:

To further enhance learning, we propose to use Bidirectional GRU (BiGRU) layers. In a BiGRU, two independent GRUs are combined in a bidirectional fashion, with one reading the input sequence in the forward direction. Conversely, the other reads the same sequence in the backward direction. The hidden states from each GRU are then concatenated, as shown in the following:

Two BiGRU layers (32 hidden units each) are stacked on top of the previously mentioned convolutional layers. The two BiGRU layers are interleaved by a conventional dropout and a recurrent dropout, both with a dropout rate of $0.1$, to further mitigate overfitting. The output of the last BiGRU is finally sent to an output neuron, with a sigmoid activation attached. Since the problem to be solved can be modeled as a binary classification task (PD/HC), the overall network is required to minimize a classical binary cross-entropy loss function. Training was done using back-propagation with the Adam optimizer and a learning rate of 0.001 on randomly sampled mini-batches of size $16$. The overall combined Convolutional-BiGRU model is illustrated in Figure 3.

Table 2 summarizes the mean accuracy values reported by the proposed model by varying the feature set given as input. Excellent performance is observed in the overall derived feature set, including kinematic as well as pressure features, calculated from the raw input acquired by the tablet. The highest predictive potential achieved a mean accuracy of over 90% in almost all cases. In contrast, the overall raw feature set reported the lowest classification rates. Not surprisingly, the kinematic features, which contribute most to the aforementioned derived feature set, exhibit the top second accuracy for all tasks among the individual feature groups. These results highlight the effectiveness of these features in capturing the impairments Parkinsonian patients have as they typically do not write with the same constancy as healthy subjects, showing a lower writing speed, with continuous acceleration peaks, e.g. [28, 27]. Significantly lower results, on the other hand, are obtained with pressure features, if considered alone. Patients generally apply less pressure on the writing surface; moreover, the pressure signal assumes erratic values due to muscular difficulties [40]. However, pressure is generally considered controversial in the literature, especially from the perspective of signature verification [29, 16], as results differ among studies. Another interesting observation is that pen inclination resulted in relatively better performance, with mean accuracy of more than 80% in two cases. The pen angle information is typically discarded in most related studies. The present results indicate that pen inclination can also be exploited in addition to kinematic and pressure information to further enrich feature representation. It is important to recall that all of these features are first fed to a series of convolutional layers, so the final set of sequences that is provided to the recurrent layers is expected to be a rich and robust representation of the discriminating attributes between PD subjects and healthy controls. These findings also corroborate the hypothesis that sequence learning may be preferred to holistic approaches for PD detection through the dynamics of handwriting.

Further considerations from Table 2 can be drawn by looking at the results obtained task by task. In general, the different feature sets agree that lll, les les les and the sentence are among the most discriminating tasks. No-sense words composed of one or more character repetitions have been used frequently for PD assessment, e.g. [8, 43], showing the impairment of Parkinsonian patients in fine motor control during loop-like movements. Indeed, PD patients may produce slower and more irregular movements; moreover, they may write letters in a more segmented fashion, showing micrographia over time when writing. Recently, Senatore and Marcelli [41] found that Parkinsonian writing during a familiar l-shape movement is characterized by a lack of fluency, slowness, and abrupt changes of direction. These difficulties support the hypothesis that the fine-tuning of the motor plan involved is deteriorated due to PD while executing a writing task.

The importance of the sentence task, already observed in [18], is also confirmed in our experiments. In fact, writing a long sentence can require a greater cognitive load, particularly a high degree of simultaneous processing. Therefore, it can increase the effects of the disease on handwriting. The high degree of simultaneous processing is due to several reasons, including the involvement of linguistic skills, attention, and memory. Producing loop-like movements and writing a sentence offers the opportunity to better evaluate the motor plan between one character or word and the next. In fact, a hesitation or pause between two characters or words can highlight the need to re-plan the writing activity. Conversely, fluid writing reveals the presence of early motor planning [8, 14]. In particular, a sentence allows one to capture a large number of in-air movements between components; conversely, a single word could be written without leaving the pen from the writing surface [19].

Another observation concerns the spiral task. As mentioned above, the task was undertaken without any significant impact on classification in previous studies, e.g. [18]. Instead, similar to what we previously observed [15, 32], the Archimedes spiral task has achieved high classification accuracy here for almost all feature sets. This reinforces the clinical validity of the task, as clinical experts commonly use it for screening for early signs of PD. One reason may be due to the time it takes to complete the task, as spiral drawing requires continuous on-surface strokes in all directions and, therefore, can better capture changes in the dynamics of handwriting in all directions.

In Table 3, we report the classification performance of the best performing feature set, i.e. derived features, in terms of AUC, sensitivity, and specificity. In general, high values are obtained for all metrics and all tasks, confirming the applicability of the proposed method. There is usually a trade-off between sensitivity and specificity. In the present work, the method appears to be slightly biased in favor of specificity. This suggests that a screening test based on our tool will be better at correctly classifying healthy subjects. To further validate the proposed method, we also illustrate (in Fig. 4) the ROC plots for each of the eight tasks where the highest true positive and lowest false positive rates validate the robust discriminating power of the proposed model.

To further establish the effectiveness of the proposed method, a comparative analysis with state-of-the-art approaches aimed at detecting PD through the dynamics of handwriting is presented in Table 4. It should be noted that, during the comparison, it was ensured that the same evaluation metric (mean accuracy) and the same validation scheme (10-fold cross-validation) were used. Indeed, this is the evaluation protocol usually adopted for PaHaW. Furthermore, since different classifiers (support vector machines, random forests, etc.) were used in these studies, reporting different classification accuracies, for a fair comparison, we report here only the best results of these studies for each task.

The first baseline selected for comparison consists of the traditional, hand-crafted dynamic features proposed by Drotár et al. [19]. In that work, the horizontal and vertical components of the pen position were segmented into on-surface and in-air strokes as a function of the button status value. Based on this segmentation, kinematic (displacement, velocity, acceleration, and jerk), spatio-temporal (on-surface time) and pressure features were derived. This feature extraction step resulted in either a single-valued feature or a vector feature. For all the resulting vector features, the following basic statistical measures were calculated: mean; median; standard deviation; 1st percentile; 99th percentile; 99th – 1st percentile. The overall feature vectors were then fed into traditional statistical classifiers, after keeping the most discriminating subset with an a priori selection of features made on the overall dataset before cross-validation. It is worth noting that supervised feature selection strategies should be nested within the cross-validation iterations, so that the most discriminating features are chosen based only on the training set, while the test set is kept aside. Resorting to an a priori selection of features over the entire dataset accidentally introduces a bias in the classification process which may lead to overoptimistic performance. This is well-known in the machine learning community; see, for example, [23].

The second baseline against which we compare the proposed method is the extension of the previously mentioned features proposed by Impedovo [25]. The author has enriched the conventional feature set with additional features obtained by applying the Sigma-Lognormal model, the Maxwell-Boltzmann distribution and the well-known Discrete Fourier Transform to the velocity profile of handwriting. Based on these features, significantly better results were obtained. However, it is worth pointing out that the author used the same non-nested validation scheme previously applied by Drotár et al. In fact, the author moved from their result as a baseline, then improved it. This means that, while remarkable, the provided contribution still suffers the same feature selection bias as the study of Drotár et al. [18].

The third baseline consists of the results we obtained by replicating the experiments carried out by Drotár et al. [18], in which, in order to uncover hidden complexities of handwriting, features based on Shannon and Rényi entropy, signal-to-noise ratio, and empirical mode decomposition were also computed [5]. In the work cited, we used a nested feature selection in which the most discriminating features were chosen based only on the training set at each cross-validation iteration. In the work, we showed the detrimental effect on classification accuracy caused by inadvertently introducing the feature selection bias into the machine learning workflow.

Finally, the fourth baseline against which we compared the proposed method consists in the use of features automatically extracted by a 2D Convolutional Neural Network model [15]. More specifically, the well-known VGG16 deep network [42], pre-trained on ImageNet [13], was applied as a feature extractor to multiple “views” of the same static representation of the handwriting. This representation was obtained by embedding dynamic temporal information during the image generation to retain velocity/in-air information. The overall feature vectors were fed into standard statistical classifiers to achieve the final classification, after dynamically retaining features with a nested cross-validation scheme.

As can be inferred from Table 4, the sequence learning approach based on 1D convolutions and BiGRUs significantly surpasses, by a considerable margin, the previously proposed procedures based on traditional dynamic features and 2D convolutions on the same dataset. This confirms the effectiveness of the proposed method to serve as a candidate solution for real use in a clinical setting. It is also worth noting that the feature selection bias problem does not apply here. Interestingly, all related works generally agree that the sentence task is among the most discriminating. Conversely, as noted earlier, the spiral task generally reports relatively lower performance when traditional dynamic features are used.

We also report the results of ablation studies we carried out to choose the model architecture. These results are summarized in Table 5 and Table 6. It is worth mentioning that these results were obtained by feeding the models with the derived feature set. The first ablation study (Table 5) was conducted to observe the performance of different RNN-based models, which served as a baseline for the outcome of the second ablation study. The features were fed directly into the recurrent units without any convolutional layers. It was observed that the Bidirectional GRU-based model outperformed the other RNN variants on each of the tasks.

The second ablation study (Table 6) mainly concerned the evaluation of the effectiveness of jointly exploiting convolutional and recurrent units. It can be observed that applying convolution on the sequences provided as input, before feeding them into the BiGRU layers, significantly improves classification accuracy. Moreover, it is worth pointing out that, due to the sub-sampling of the time sequences obtained using different stride size (greater than one), the training complexity of all RNN models is also reduced. Overall, these findings further validate our hypothesis that time-based handwriting sequences contain unique patterns that can be enhanced by convolution and identified by Bidirectional GRUs for PD classification.

To validate the generalization capacity of our approach, we ran a series of experiments on NewHandPD. This dataset was not seen during the configuration of our system. Therefore, the best configuration found has been used here.

Specifically, the experiments with NewHandPD have been performed using an experimental protocol similar to that used in [38]. This included using 65% of the data for training, 10% for validation and 25% for testing. The results obtained have been averaged after repeating the experiment for 20 runs. It is worth noting that we applied the same data preprocessing suggested in [38], which consisted of removing outliers by cutting off values below the 5th percentile and above the 90th percentile on each channel. Moreover, a $z$-score normalization was applied.

The experimental results are provided in Table 7 for each type of exam. Similar to previous results, we reported performance in terms of AUC, sensitivity, specificity and accuracy. Competitive results have been obtained, especially for the spiral-based exam. In addition, the specificity results are consistently greater than sensitivity in all cases. These two effects give consistency to our method as performance with PaWaH also showed similar findings.

Furthermore, we have analyzed the previous literature with this particular database in order to contextualize our results. In [36], the authors proposed to model the time-series of the NewHandPD dataset as images. The images were designed using the six available channels as well as the temporal sequences. Then, different CNN architectures were exploited. Specifically, the authors experimented with spirals and meanders. Among the different configurations studied, their best results are shown in Table 8. They correspond to a network pre-trained on ImageNet, accepting $128\times 128$ images and using 75 % of the dataset for training. The authors also reported the results obtained using other classifiers, such as Optimum-Path Forest.

A similar approach was presented in [34]. Again, several CNN architectures were investigated to discriminate between healthy controls and PD patients. In this case, all exams available in NewHandPD were studied using samples from 20 healthy controls and 14 PD patients. The results shown in Table 8 were achieved with 50 % of the specimens for training and using the ImageNet-based network. As a step forward, the authors also presented the results obtained by combining all the exams to achieve a single classification.

For the most relevant comparison, we have selected the work of [38], in which stacks of Bidirectional Gated Recurrent Units were employed with an attention layer on top. The authors introduced a bag-of-sampling concept for selecting samples of signal sequences provided in the NewHandPD dataset. The results show that this approach led to better classification outcomes compared to previous studies (Table 8). The experiments were performed using 65% of data for training, 10% for validation and 25% for testing. Instead of using the entire database, the authors used 25 control subjects and 14 patients.

The overall results of our proposed model on the NewHandPD data with a similar experimental protocol show a significant improvement in classification when compared to the results of [36] and [34]. When compared with [38], it is observed that our method has improved results when the spiral is used for classification, while in the case of meanders, our method behaves comparatively with [38]. It is also noteworthy that our results are computed using all the samples, i.e. 35 healthy and 31 diseased subjects, provided in the NewHandPD database.

Finally, it is worth pointing out that all the results shown correspond to the best configurations studied in each paper for the NewHandPD database. In our work, we have used this database only to demonstrate the generalization capacity of our approach.