Deep Learning Based Walking Tasks Classification in Older Adults using fNIRS

The study involved a total of $n=451$ community dwelling older adults in Lower Westchester county, NY of age 65 years and older (76.16 $\pm$ 6.67, 223 females) who were originally enrolled in a longitudinal cohort study entitled “Central Control of Mobility in Aging” (CCMA) [4, 25]. Recruitment procedures started with the identification of potential participants from population lists and then conducting a structured telephone interview to obtain verbal assent, assess medical history, mobility and cognitive functioning. Participants with significant loss of vision and/or hearing, inability to ambulate independently, current or history of severe neurological or psychiatric disorders, and recent or anticipated medical procedures that may affect mobility were excluded from the study. Individuals who agreed to participate in the study, fell into the inclusion/exclusion criteria and passed the phone interview were invited to two annual in-person study visits each lasting around 3 hours at the research center at Albert Einstein College of Medicine, Bronx, NY. During these visits, participants received a structured neurological examination and comprehensive neuropsychological, psychological, functional, and mobility assessments. Functional brain monitoring using fNIRS during the STW and DTW protocol was completed in one session. Cognitive status was determined at consensus diagnostic case conferences [47]. Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) was used to characterize overall level of cognitive function [48]. The sample was relatively healthy (Global Health Status mean score = 1.62 $\pm$ 1.09) and in the average range of overall cognitive function (RBANS mean Index score = 91.77 $\pm$ 11.71). The work described in this manuscript has been executed in adherence with The Code of Ethics of the World Medical Association (Declaration of Helsinki) and the APA ethical standards set for research involving human participants. Written informed consents were obtained at the first clinic visit according to study protocols approved by the Institutional Review Board at Albert Einstein College of Medicine, Bronx, NY (Protocol #2010-224; Date: 03/03/2022).

The task protocol used in this study involved two single tasks and one dual-task conditions presented in a counterbalanced order using a Latin-square design to minimize task order effects on the outcome measures. The single task conditions were 1) single-task walking (STW) and 2) the single task alpha cognitive interference task (STA). In STW condition, participants were asked to walk at their “normal pace” around a 4 $\times$ 20 foot electronic walkway (Zenometrics system with Zeno electronic walkway using ProtoKinetics Movement Analysis Software (PKMAS), Zenometrics, LLC; Peekskill, NY). In the Alpha condition participants were asked to stand still on the electronic walkway while reciting alternate letters of the alphabet out loud (A, C, E…) for 30 seconds. In the dual-task walking (DTW) condition, participants were required to perform the two single tasks at the same time by walking around the walkway at their normal pace while reciting alternate letters of the alphabet. Participants were specifically asked to pay equal attention to both the walking and cognitive interference tasks to minimize task prioritization effects. In both STW and DTW conditions participants were asked to walk on the instrumented walkway in three continuous loops that consisted of six straight walks and five left-sided turns. The duration of each task condition varied depending on the individual’s walking speed. Reliability and validity for this walking paradigm have been well established [49].

During the fNIRS data collection procedure, first the fNIRS sensor was placed on the forehead of the recruited participants. A standardized sensor placement procedure based on landmarks from the international 10–20 system was implemented [52, 53] where middle of the sensor was aligned with the nose horizontally and the bottom of the sensor was placed above the eyebrows vertically. Testing was conducted in a quiet room. Participants wore comfortable footwear and performed the task protocol with the fNIRS sensor attached to their forehead during the overall data collection period.

First, visual inspection was performed on individual data from all voxels to identify and eliminate the ones with saturation, dark current conditions or extreme noise. Then to eliminate spiky type noise, wavelet denoising with Daubechies 5 (db5) wavelet was applied to the raw intensity measurements at 730 and 850 nm wavelengths as proposed in [54] and widely applied in fNIRS studies [55]. The artifact-removed raw intensity measurements were then converted to changes in HbO2 and Hb using modified Beer-Lambert law (MBLL) [20, 30, 56]. In MBLL, previously published values for conversion parameters i.e. wavelength and chromophore dependent molar extinction coefficients ($\epsilon$) and age and wavelength adjusted differential pathlength factor (DPF) were used [20, 30, 57]. Finally, we applied Spline filtering [58] followed by a finite impulse response low-pass filter with cut-off frequency at 0.08 Hz [20, 59] to HbO2 and Hb data separately to remove possible baseline shifts and to suppress physiological artifacts such as respiration and Mayer waves.

Data epochs corresponding to each task condition, STW, STA and DTW, were extracted to be used in further processing for feature extraction and machine learning model generation for automatic activity classification. fNIRS data acquisition and the electronic walkway system for gait analysis were synchronized using a central “hub” computer with E-Prime 2.0 software where time stamps of start and end points for each baseline and task condition were marked and recorded [27, 28, 13, 14, 15, 16, 17, 18, 19, 20]. In order to correctly extract the data epochs during the exact walking task execution periods, a second level processing time synchronization method was implemented. The HbO2 and Hb data epochs corresponding to time interval between the first recorded foot contact with the walkway until the end of the 6th and final straight walk algorithmically determined by PKMAS as previously described in [26] were extracted for STW and DTW conditions. Finally, proximal 10-second baselines administered prior to each experimental task were used to determine the relative task-related changes in the extracted HbO2 and Hb data epochs for each of the task condition using the previously described baseline correction method (subtracting the average value of the proximal baseline region data from the following task epoch data) [27, 28, 13, 14, 15, 16, 17, 18, 19, 20]. We then used HbO2 and Hb data epochs in DTW, STW and STA tasks in further feature extraction and machine learning model development to automatically classify these three tasks in this work.

In this sub-section, we first show the statistical analysis across and within subjects, then present the feature engineering including input formulation for subsequent deep learning algorithms.

Additionally, we investigate the distribution of Hb and HbO2 values under the three task conditions as provided in the histogram plots in Fig. 4. For Hb levels, all the three task conditions exhibit a Gaussian-like distribution where the averages are around negative values close to 0, suggesting decreases relative to baseline conditions. DTW condition shows higher standard deviation than the rest of the two single tasks, suggesting more individual variability in this condition. For HbO2, distribution of STW is still showing Gaussian-like pattern with average around the positive values close to 0. However, for DTW and STA, the distributions have shifted to more positive values suggesting more cognitive activations relative to baseline in these task conditions as compared to STW. Comparatively, levels in DTW show larger shift than STA. Such distribution differences between Hb, HbO2 and HbO2 - Hb levels inspire us in this study to use them as additional features for the machine learning model.

Although there are 16 voxel locations, due to limitations on data collection with excessive artifacts that cannot be cleaned, there are missing fNIRS data of one or a few voxel locations in some samples. We perform a screening and accept samples with no more than 3 missing voxel locations and any samples with missing numbers beyond 3 is regarded as invalid data and thus not used in training or inference. The number of valid samples in the dataset after removing is $n=1216$. For the missing data we use interpolation to reconstruct the data into an image of size $3\times 16\times L_{i}$.

Since the dataset is relatively small compared with general computer vision datasets with usually more than tens of thousands of images, we use pre-trained deep learning models rather than training models afresh to prevent over-fiting. Additionally, due to the difference on task completion time, the dimensions of the images are not consistent. Therefore, we perform further image processing to conform the input image dimensions with the input of the deep learning models that are usually pre-trained with the ImageNet dataset [60], i.e., $3\times 224\times 224$. Although such processing or resizing of inputs are common in deep learning, how the images are resized into $3\times 224\times 224$ can potentially have impact on the learning performance [61]. Note that, for each sample, data beyond first 224 points are truncated while data with less than 224 points are padded with 0. We intentionally avoid row-wise interpolation because we want the fNIRS data to keep the information of task completion time and pace of each subject which can be an important marker.

In order to find the optimal processing configuration, we sweep across 3 different parameters which are two commonly used resizing techniques for deep learning model inputs [61], which enables us 5 different configurations. The 3 parameters are visualized in Fig. 6 and also explained below:

• •

  Interpolation mode: Interpolation is to insert data between each original data to enlarge the input size into $3\times 224\times 224$. The available options are bilinear or bicubic interpolation.

• •

  Corner alignment: This parameter decides whether to force corner alignment during interpolation. If true, the levels from the first and last voxel locations will be used as corner so that no interpolated values will appear as corners of the processed image.

• •

  Interleave repeat: As we have 16 voxel locations, we are able to interleave repeat the data of each location by 14 times to achieve the required 224 dimension (row-wise) as an alternative for interpolation. Interleave repeat will duplicate exactly the data row-wise, which will reflect as the clear boarders between each voxel locations visually as illustrated in Fig. 6.

In summary, after feature engineering, we are able process and enhance the original samples with fNIRS levels of missing data and inconsistent dimensions into images in the shape of $3\times 224\times 224$, which is ready for the deep learning models.

We leverage the state-of-the-art deep learning models for machine vision and/or image classification. We first train the models afresh, i.e., without any pre-training. However, since the scale of the fNIRS dataset is rather small, the model easily overfits: the accuracy on training set achieves more than 90% while the inference set is only around 70%.

This is a common issue for other applications from medical imaging in the bioinformatics domain as the clinical data are usually much less than large general image datasets. Thus, for domain specific application with proprietary datasets, the concept of transfer learning [62] is usually in favor: the model is first pre-trained using the general and public image dataset and then fine-tuned on the proprietary dataset. In this work, using ResNet as an example, we obtain open-sourced, publicly available ResNet model which is trained using the ImageNet dataset which has 1000 classes. We then change the dimensionality of the last classifier (the fully connected layer) from 1000 to 3 to match the three task conditions. We fine-tune over this model by using the processed images to fit the model on the fNIRS dataset for classifying the three task conditions.

We implement our deep learning model using various architectures that are basically in two families: deep convolutional neural networks and vision transformer attention networks. Model details and configurations are introduced in the experimental results section Sec. 4.

As a comparative study, we also implement several baseline machine learning models, including traditional machine learning models of decision tree, random forest (with 25 trees), and k-nearest neighbors (k = 5) which are implemented via Scikit-Learn package [70].

We can first observe that deep learning models are able to out-perform all the baseline traditional learning algorithms by at least 10% on the inference set. Particularly, decision tree and random forest are extremely over-fit on training data with higher than 99% accuracy while having poor performance on inference set. K-nearest neighbors can hardly learn efficiently as the accuracy on training set is only around 77% which is quite lower than all the other models.

Some of the deep learning models also experience different degree of overfit. Particularly for larger models such as VGG, ResNet and ViT-Base, their accuracy on training set is mostly over 94% except for VGG. Smaller models like MobileNetV2, TinyNet and EfficientNet are usually less overfit as their accuracy is 90% – 93% on training set.

The top three models according to accuracy on inference set are: ResNet-18, VGG-13 and TinyNet-E. All the three models can achieve accuracy near 81%. We use these three models for the following ablation study and efficiency analysis.

We also provide insights on the three models for their cost and overhead including number of parameters, model size, time for model fine-tuning and inference as well as the throughput (samples per second) in Table 3. Time and throughput data are obtained with NVIDIA Tesla P100 GPU.

ResNet-18 and VGG-13 use relatively larger model size and show lower throughput. Particularly for VGG-13, although the model is quite large with around 130M parameters, it does not show more competitive accuracy than the rest. For TinyNet-E, since the classifier layer (fully connected layer) output is reduced to 3, the model size becomes drastically compact and the throughput nearly triples the rest two models. However, based on our experiment, TinyNet-E requires more epochs to achieve comparable accuracy, thus even if it is smaller in model, it takes more time than ResNet-18 to fine-tune.