Predicting dominant hand from spatiotemporal context varying physiological data

Smartwatches are getting higher technology sensors and processing units for increasingly affordable prices. Sensors commonly include GPS, gyroscopes, pedometers, temperature, blood oxygen (SpO2), and advanced heart monitoring. The available data enables gesture recognition, physical activity identification, and estimation of relevant health metrics. These wearable devices are now a popular alternative to easily monitor wellness information, such as energy expenditure, sleep quality, heart rhythm assessment, and stress levels.

The proposed two-hand experiment takes advantage of those characteristics to develop the dominant hand prediction algorithm. The subject wore a Fitbit Sense device on each wrist following its recommended tightness and location for all-day wear with one finger width above the wrist bone, see Figure 1. Given the long 2-week time frame, the smartwatches were removed to keep consistent measurement conditions. This was done for recharging the devices, a daily cleaning procedure, and also to prevent interaction with water.

After the initial setup, the settings were kept with default values. All the available features were turned on, and the devices had an active Fitbit Premium subscription during the experiment. Each wristband was connected to a different smartphone due to a one-device restriction for the official application. Both device pairs were kept reasonably close to ensure a consistent Bluetooth link during the experiment.

Lastly, it is important to highlight that the subject was instructed to keep routines unchanged. Other than wearing the pair of smartwatches, keep them charged, and clean. The data should record typical unconstrained real-life conditions for a graduate student without mobility restrictions.

A simple and short survey was designed to easily gather detailed information on the subject’s context. The user was prompted to make only between 1 to 4 selections among different options of interest. Those options included 10 common locations, 16 activities, 5 levels of physiological arousal, and a flag to indicate when the device was removed. The survey was implemented using Google Forms as shown in Figure 2.

The suggested procedure was filling out the survey before any change of activity or location. A shortcut access was added on the home screen of the subject’s main smartphone to remember the self-report task. However, the subject remarked serious difficulties to keep on the self-report during the experiment. It was difficult to establish the routine, the task was easily forgotten even with an expressed high level of engagement with the experiment. Unreliable internet connection and slow smartphone responses further contributed to the problems.

The survey demanded unrealistic commitment under a highly dynamic real-life routine. So, the subject completed some information using the survey, but also manually included additional entries to complete the spreadsheet at the end of each day. The manual updates considered the inspection of additional sources of information to increase accuracy on the context labels. That includes using GPS information from Google Timeline records, heart rate, and step count from the Fitbit dataset. During the experiment 256 context updates were completed using the survey, and 241 additional updates were included manually.

It is worth clarifying that the purpose of the survey is just to facilitate the research process. However, a fully practical implementation would require the extraction of contextual information from the available sources. For instance, location can be automatically extracted from GPS information, and activity can be estimated from the numerous sensors on both smartphone and smartwatch devices.

The process starts using the official data export tool from the Fitbit website to download the most detailed raw data available. Files for heart rate, step count, calories, and altitude are stored in separate folders for each hand. The manufacturer encodes the information in multiple JSON-formatted files.

A python code with inspiration on [13] is developed to convert the JSON files into a single CSV file for each hand. The CSV files are further processed using Matlab to remove unnecessary information, adjust nonuniform date-stamps, and re-sampling to a consistent 1-second interval dataset. Missing information was filled using linear interpolation for heart rate, and zero value for the remaining variables. Clean and consistent datasets for each hand are stored as timetables on MAT files.

For the self-reported contextual information. The spreadsheet containing the data is directly processed using Matlab. A timetable structure is also created and string data types are converted into categorical. A MAT file is saved with a re-sampling of a 1-second interval consistent with the physiological data. Another one is saved with a greater re-sampling under the window interval used for feature calculation. A new time categorical column is created to divide time into four more meaningful ranges (Noon, Morning, Afternoon, and Evening).

Time synchronization is then performed using the Matlab timetables to merge both physiological and contextual data with the uniform 1-second intervals. The physiological data is further cleaned eliminating time frames where the Fitbit devices are removed. Finally, the corresponding dominant and non-dominant hand labels are assigned to the clean, uniform, and organized datasets ready for feature extraction.

The objective on dominant hand prediction was clearly defined only after the experiment design and its data collection. So, some of the collected data is dropped as irrelevant given the context of this problem. For instance, altitude from the Fitbit data, and subjective physiological arousal form the self-report survey. The confidence for each heart rate sample was also omitted given the similar distribution of its values for both dominant and non-dominant hand, see Figure 3.

The features for the remaining data were calculated using different window sizes of 1, 5, 10, 20, and 40 minutes. For heart rate 3 frequency-based, and 17 time-based statistical features are calculated. The cumulative sum was the only feature considered for step count and calories. The 3 categorical features were not further processed and completed a set with 25 features in total.

The experiment was extended two additional weeks to test two different settings available on Fitbit devices. The first one with both devices on default non-dominant hand setting. The second one with the devices configured accordingly to the dominant or non-dominant hand setting. The features are calculated for both conditions and stored in separated files.

The Minimum Redundancy Maximum Relevance (MRMR) Algorithm, available on Matlab, is initially used for feature selection. Following an heuristic approach, it provides significant scores for classification tasks with both categorical and continuous features. The results show low relevance for most of the heart rate and categorical features.

For the example shown in Figure 4, features with top-five MRMR scores include calories, step count, and heart rate values for range, minimum, maximum and number of peaks. The results vary depending on the window size, but also on the Fitbit setting being used, something evidenced on the example shown. However, it is important to remark that most categorical features got low or zero scores on almost every tested condition.

Notwithstanding the MRMR scores, categorical features contain some valuable contextual information such as activity, location, and time. From literature, see [9], it is clear that certain activities are strongly connected with dominant hand preference. Therefore, a further feature filter accounting to context and problem knowledge is proposed.

The context-based feature filter eliminates windows potentially providing indistinguishable data for dominant prediction. That includes activities involving reduced body movement, such as sleeping, movies, or meetings. Physiological data suggesting similar conditions, like zero step or zero calories counts. Also, including means to filter windows with specific activities or conditions facilitating dominant hand prediction.

The Matlab Classification Learner toolbox was used to evaluate a broad variety of models. It includes up to 32 different model configurations such as decision trees, discriminant analysis, logistic regression, naive Bayes, support vector machines (SVM), k-nearest neighbor (KNN), kernel approximations, ensemble alternatives, and neural networks. The models supporting both categorical and continuous features are strongly limited, but most of the tests included continuous features only. The main configurations across all the different evaluations were set to a 5-fold cross-validation method and 10% proportion for the test dataset. Access to the data and codes is available on the following link: https://drive.google.com/drive/folders/1kdaSetBQWjWsV2pbKZ-Y41dZirmyWskS?usp=sharing