Health Guardian: Using Multi-modal Data to Understand Individual Health

Depression is a prevalent mental health condition that affects many individuals worldwide. In 2020, an estimated 21.0 million (8.4%) adults and 4.1 million (17.0%) adolescents aged 12 to 17 in the United States alone, experienced at least one major depressive episode [21]. This condition can impact a person’s feelings, thoughts, and ability to perform daily activities, leading to persistent feelings of sadness, reduced interest in hobbies, and hopelessness lasting more than two weeks. In addition, depression can also cause physical symptoms such as changes in appetite and sleep patterns, fatigue and difficulty concentrating, thus affecting the individual’s capacity to work, sleep, study, and eat.

Validated self-report measures such as the Patient Health Questionnaire (PHQ-8) [15], the Beck Depression Inventory (BDI) [22] and the Depression Anxiety Stress Scales-21 (DASS-21) [23] are commonly used for screening and assessing the severity of depression, guiding recommended treatments, and monitoring symptoms and recovery. However, traditional methods of administering these assessments are limited by the need for in-person, pencil-and-paper administration at a clinic, which can reduce the frequency at which individuals can be screened and monitored.

To address this issue, the IBM Digital Health team has developed a digital, text-based microservice using PHQ-8 depression questionnaire as a framework. The PHQ-8 consists of eight items that align with the diagnostic criteria for major depression disorder in the Diagnostic and Statistical Manual of Mental Disorders Fourth Edition (DSM-IV) [15], which is widely used for depression diagnosis and assessment. This digital microservice can increase access and the frequency at which individuals with mental health disorders can be screened and monitored.

The PHQ-8 questionnaire is delivered to users via the Health Guardian mobile application (Figure 3a.i). Users can select the frequency of each depression symptom they have experienced in the past two weeks, and upon submission, the dataset is parsed into a text-based response in .json format (Figure 3a.ii), and uploaded to the clinical task manager (Figure 3b). The analytic worker processes the structured data, extracts the selected responses from each question (Figure 3c.i), and calculates the total score (Figure 3c.ii). The results are sent back to the mobile application via an API-gateway (Figure 3a.iii), where users can view and track their responses over time. The results are stored as structured data in a datastore, which can be connected to a clinician dashboard or other data rendering interfaces. These tools would enable clinicians or psychiatrists to observe changes in individuals’ symptoms and treatment progression and make informed decisions regarding intervention strategies.

The PHQ-8 questionnaire consists of eight items that evaluate symptoms of depression, with each item scored from 0 points (indicating ”not at all” present) to 3 points (indicating ”nearly every day” present). The total score can range from 0 to 24, with scores categorized as follows: no significant depressive symptoms (0 to 4), mild depressive symptoms (5 to 9), moderate (10 to 14), moderately severe (15 to 19), and severe (20 to 24) [15].

Currently, the microservice provides a score for each completed PHQ-8 questionnaire (Figure 3a.iii), and stores the data longitudinally with time. Leveraging AI/ML models, it is possible to analyze the responses collected from multiple PHQ-8 questionnaires to derive more valuable insights. By considering longitudinal data, a deeper understanding of an individual’s depressive symptoms and their progression can be obtained, enabling personalized and effective interventions.

Clinically, a variant of the PHQ-8, called the PHQ-9 questionnaire is used. The PHQ-8 differs from the PHQ-9 in that the PHQ-8 excludes the last question in the PHQ-9 which asks about thoughts of death and self-harm [15]. The PHQ-8 has shown comparable ability to PHQ-9 in diagnosing and assessing depression among various populations [24, 25, 26]. Since the question about suicidal ideation is concerning when real-time psychiatric intervention or further suicidal evaluation are not provided [25], PHQ-8 may be a better alternative for depression screening when appropriate suicidal evaluation is not provided in the study or research.

Advancements in computer vision and deep learning have enabled the development of video-based techniques for contact-free and passive evaluation of Parkinson’s Disease (PD) symptoms during daily activities. PD is the second most common progressive neurodegenerative disease that affects 2-3% of the population over 65 years of age [27]. Its hallmark characteristics are motor symptoms, such as bradykinesia (BRADY), tremors, rigidity, posture instability and gait disorders (PIGD), along with non-motor symptoms such as olfactory dysfunction and sleep disorders. Although no cure exists for PD, dopamine replacement therapy can mitigate symptoms and improve patients’ quality of life.

Managing the various types of PD requires that neurologists comprehend the severity of symptoms and extent of motor fluctuations, which may require changes in medication timing and dosage. In-person assessments at a clinic done once or twice a year, follows protocols specified in the Unified Parkinson’s Disease Rating Scale (UPDRS) and provide neurologists with information on the severity of symptoms, motor fluctuations, and medication efficacy.

In order to gain insights into changes in motor symptoms between clinical assessments, clinicians have asked PD patients to provide daily self-assessments or use wearable sensors to track mobility symptoms. However, recall bias limits the usefulness of self-reports, and compliance adherence issues can affect the deployment of wearable sensors. To address these challenges, IBM’s Digital Health team has developed a video-based method that takes a short 20-30 second video containing sit-stand movements as input to predict the UPDRS subscores, such as BRADY and PIGD [11].

The sit-to-stand microservice is provided on the Health Guardian mobile application. Using the mobile application, the user is prompted to take a short video (~20-30 s) of themselves cycling from a sitting to a standing position several times (Figures 4a.i and 4a.ii). The user uploads the video in .mp4 format from the mobile application to the clinical task manager (CTM) where the raw data and metadata of the file are stored in a database and cloud object storage (Figure 4b). The video file is processed by the analytic worker for this microservice to calculate the UPDRS scores, and generates a torso motion graph depicting the sit-stand movements and associated hesitations. The UPDRS scores and torso motion graph are sent back to the mobile application via an API-gateway (Figure 4a.iii).

The analytics for the sit-to-stand microservice involves several steps and is described in detail in [11]. Briefly, a short input video sequence (Figure 4c.i) is processed by a human detector to extract the video frame-by-frame (Figure 4c.ii). The resulting data are then passed to a 2D pose estimation model that predicts the coordinate locations of human joints in 2D image space (Figure 4c.iii). Next, a 3D pose model utilizes the 2D pose information to predict joint locations in 3D Cartesian space (Figure 4c.iv). Finally, the 3D pose information is utilized in three different ensemble combinations that incorporate Hierarchical Convolutional Network (HCN) [28], Spatio-Temporal Graph Convolutional Network (ST-GCN) [29] and/or Convolutional Networks (CNNs) such as ResNet50 [30] (Figure 4c.v). The UPDRS score is predicted from the model, and graphs such as real-time torso motion can be generated from the processed data (Figure 4c.vi).

In two separate clinic visits, video clips of sit-stand motions from 35 subjects were captured and used to validate the analytics of the sit-to-stand microservice. This evaluation was part of a larger UPDRS assessment supervised by a neurologist, who was also assigned to score each task. The study demonstrated that it is possible to predict BRADY and PIGD scores from a short sit-stand video clip, with F1-scores, a measure of a model’s accuracy, from the AI models performing better than the results from the two clinician video raters [11].

The Timed Up and Go Test (TUG) is a clinical assessment tool used to evaluate balance and gait in everyday tasks such as sitting, standing, walking and turning. It is commonly used to examine functional mobility in older adults (aged 65+) who may be frail and have a history of falls [31]. The test involves standing up from a chair, walking 3 meters (10 feet), turning around, walking back to the chair, and sitting down again. The time taken to complete the test, measured in seconds, is strongly correlated to the level of functional mobility.

Research has shown that older adults who took 13.5 seconds or longer to perform the TUG were at higher risks of falling, with a positive prediction rate of 90% [32]. Additionally, studies have identified cutoff scores of 11.5 seconds in Parkinson’s disease patients [33], and 14 seconds for patients with strokes [34] as indicating increased fall risks.

The TUG test can also be used to monitor disease progression and changes in the quality of life for patients with mobility impairments as a result of specific diseases, as demonstrated in studies of patients with Parkinson’s disease [35] and patients recovering from hip and knee arthroplasty [36]. Although TUG is a valuable clinical tool, its limitations include the need for in-person assessment and the results are only measured as a snapshot in time.

To adapt the TUG test to a digital health platform, IBM Research scientists have developed an automated TUG prediction AI model that utilizes accelerometer data from a wrist-worn watch. This model was transformed into a microservice using the standard HG worker and API-gateway template [9].

To use this microservice, the subject first connects the HG mobile application to a wearable watch (e.g. TicWatch or Samsung Galaxy Watch series) that has the HG companion application installed. Next, the user selects the TUG microservice on the HG mobile application (Figure 5a.i), which prompts the user to press the ”Start Sensor” button on the watch. The user then walks for at least 30 seconds until the watch buzzes to signal completion. The raw accelerometer signals are encrypted and sent in .json format via Bluetooth to the mobile application, which is then uploaded to the HG backend services for further processing (Figure 5b). The analytic worker processes the accelerometer data and calculates the TUG prediction score as well as the average TUG scores (if multiple walks were recorded in a given day) (Figure 5c). The results are then reported back to the mobile application via the microservice’s API-gateway (Figure 5a.iv).

The analysis framework for the TUG prediction model using wrist-worn accelerators is described in detail in [10]. Briefly, the raw accelerometer data from the wrist-worn accelerator is pre-processed to identify walking episodes and to calculate step duration using step detection (Figure 5c.i). From the step durations and time difference between consecutive step durations, 20 statistical features are extracted for each identified walking episode using a Random Forest model (Figure 5c.ii). The predicted TUG score is derived from these statistical features, with the 25th and 5th percentile of step duration and the mean step duration having the greatest impact on the predicted TUG score.

To validate this model, it was applied to three datasets that contained wearable recordings of walks and TUG scores from 303 subjects, including healthy individuals, those with Parkinson’s disease, and those with mild cognitive impairment or dementia. The two public datasets used were the Long-term Movement Monitoring database (LTMM), which had subjects wear an accelerometer on their lower back, and the Gait in Parkinson’s Disease (GPD) database, which recorded accelerometer data using in-sole gait sensors. The third dataset, the Dementia Behavioral Study dataset (DBSD), was collected by the University of Tsukuba and IBM Research and used wrist-worn accelerometers to record data during in-lab walking. The validation results demonstrated that the Random Forest-based predictive model for TUG had good clinical correlation, and achieved an accuracy of 1.7 +/- 1.7 seconds, with 84.8% of the predictions falling within the minimal detectable change across all three separate cohorts [10].