Predicting Clinical Deterioration in Hospitals

More than 5.7 million patients are admitted annually to intensive care units (ICUs) in the United States [10], with an estimated cost ranging from 121 to 262 billion dollars a year [16] and a mortality rate of about 11% [15]. Unplanned admissions to the ICU are accompanied by an increase in-hospital mortality rate and length of stay. Patients with deteriorating conditions often manifest abnormalities in physiological signals and laboratory test results before becoming unstable. The belief that rapid intervention in response to warning signs might provide a better outcome for these patients gave rise to a patient safety intervention philosophy known as a Rapid Response System (RRS). RRS use a pre-determined set of criteria to identify patients at risk of imminent clinical deterioration for immediate assessment and treatment. However, systematic study of these rule-based systems has failed to show better outcomes as measured by a decrease in ICU transfers or deaths. There are a number of reasons that the current RRS are inadequate: 1) they are not sensitive enough, 2) they don’t provide sufficient lead time for medical intervention, 3) they do not continuously monitor patients, or 4) they fail to systematically monitor all patients [2].

A comprehensive, automated, timely, and accurate approach to identifying patients who are at risk for clinical deterioration could improve RRS intervention outcomes. One approach is to apply machine learning on vast historical electronic medical records (EMR) data to create state-of-the-art systems for detecting clinical deterioration; if these AI-based systems can detect patients at risk earlier and more accurately (more sensitive, no less specific), this would allow hospitals to do more timely interventions, and potentially reduce mortality and improve prognosis.

In this study, our goal is to create an automated prediction model based on real-time electronic medical records to identify patients at high risk of clinical deterioration. We use unplanned ICU transfers as a proxy for clinical deterioration. We leverage thousands of predictor variables from EMR data, rather than dozens as is common in the current RRS. We hypothesized that such complex models provide better accuracy at longer lead time, providing more time and opportunity for clinicians to act to reverse deterioration. We compare our model with the Spanish ICU without walls (ICUWW) method [1], which has been used at the same hospitals.

Figure 1 shows a conceptual architecture for building and deploying our machine learning clinical deterioration model.

Data ingestion and preparation involves extracting data from EMR system, cleaning data, and unifying different coding standards. In practice, differences among hospitals can be substantial, making it difficult to implement and deploy an effective system across multiple hospitals. The adaptation of common coding standards by hospitals, such as LOINC for lab tests and fluids in/out, International Classification of Disease (ICD) for diagnoses and procedures, and ATC for medications, has helped to improve the usability of EMR data. Despite this, hospitals may use different standards for the same data, such as NDC and ATC for medications, or transition to a newer version of a standard, such as from ICD-9 to ICD-10. A robust system has to deal with this heterogeneity before building machine learning models. Moreover, since many EMR systems are still created and updated manually, there is room for human error and data quality issues. A proper data cleaning pipeline includes preparing data for analysis by removing or modifying data that is incorrect, incomplete, irrelevant, duplicated, or improperly formatted.

Model creation and management involves a feature extraction and a SQL generation tool that prepares data for model building phase. Extracted features and the SQL generation tool are explained in details in section III-A.

Integrating machine learning solutions and big data algorithms to the current hospital IT system is a challenging task. A closed loop architecture indicates how predictive data shows up in clinical workflows. To integrate predictive models into different EMR systems at the point of care, we can use application program interface (API) to retrieve real-time scoring predictions . As soon as new data is available for a patient, patient information along with the new data are submitted to an API Gateway. API Gateway makes a call to the scoring service and generates a prediction score. Model deployment component is not currently addressed in our machine learning solution. It will be implemented for operationalizing the solution in future.

In this section, we give an overview of the data and define the prediction target. We formulate the real-time prediction of the target from the streaming EMR data as a classification problem and describe the features extracted for model learning. We also describe the model evaluation metrics and our modeling approach.

In this section, we compare our models with the ICU without walls (ICUWW) method [1], which was used at the same hospitals. Under this method, an alert is raised if any of the lab results shown in Table III are observed. We implemented these criteria for all admissions in our data, using the same set of timestamps described earlier.

Table IV shows the performance of our models and of ICUWW. For each method, we report its sensitivity, specificity, AUC, the median advance warning for true positives, and the mean number of admissions per day with an alert. For our models, these metrics were calculated on the 20% test set that was not used for model learning.

The ICUWW method has high specificity but low sensitivity; its average sensitivity over the three hospitals is only 37%. This is perhaps unsurprising: the 30 “track and trigger” scoring systems reviewed in [11] have a median sensitivity of 24% and a median specificity of 92%.

In Table IV, two settings of the score threshold are shown for each model: one with the same specificity as ICUWW and one with a specificity of 75%. At the same specificity, our models are more sensitive, have the same or greater advance warning, and raise a similar number of alerts. The models also have AUCs above 0.85, compared with 0.57–0.69 for ICUWW (these are independent of the threshold).

Even though the sensitivities are improved, they are still too low to be useful. If the model specificities are lowered to 75%, the sensitivities are increased substantially, to about 80-85%, at the cost of raising about 3 times as many alerts. Figure 5 shows the sensitivities and specificities that can be attained by our models by varying the score threshold. Our models also compare favorably with mammography, a routine screening test, which has a sensitivity of 67.8% and a specificity of 75%.

We make a final observation from Table IV: the median advance warning for hospital 3 is much shorter than the other two hospitals. We found that this is because hospital 3 sends a significantly larger fraction of its patients to the ICU, and does so sooner. If we omit admissions where the ICU transfer occurred within 24 hours of the patient being admitted, the median advance warning for hospital 3 increases to 1.5 days for ICUWW and 2.8 days for the model with 75% specificity. The average for the three hospitals becomes 2.9 days for ICUWW and 3.6 days for the model.

Figure 6 shows the actual unplanned ICU transfer rates within each predicted score quintile of the test set. The first quintile comprises admissions with scores in the bottom fifth of all scores, the second quintile comprises the next fifth, and so on. We observe a strong correlation between the predicted score and the ICU transfer rate. This shows that our models are well calibrated.

The Joint Commission (https://www.jointcommission.org/) is an independent, not-for-profit organization that aims to improve health care for the public in collaboration with other stakeholders. A panel of patient safety experts advises the Joint Commission on a set of patient safety goals called the National Patient Safety Goals (NPSGs). NPSG requires hospitals to implement systems to enable intervention from healthcare staff when a patient’s condition is worsening, which has led to the Rapid Response System (RRS) intervention philosophy. Some studies have shown that the implementation of RRS slightly reduces cardiac arrest in general ward patients but does not reduce overall hospital mortality [14].

The Modified Early Warning System (MEWS) [12] is a widely used physiologic scoring system for warning about clinical deterioration. It is based on the patient’s respiratory rate, SpO2, temperature, systolic blood pressure, heart rate, and level of consciousness. MEWS typically offers an advance warning of deterioration of 6 to 8 hours. Our benchmark ICUWW is another scoring system; it uses five lab test results. Recent systematic reviews have demonstrated that trigger alarms only marginally improve outcomes while substantially increasing physician and nursing workloads [22]. Moreover, these methods respond to triggering events that may not signal a truly deteriorating patient, they suffer from high rates of false alarms (70-95%), which results in alarm fatigue and inappropriate resource utilization.

Although such rule-based systems have been widely adopted by hospitals, multiple systematic reviews have failed to find evidence of their effectiveness. Typical calling criteria are based on physiologic measurements, nurse assessments, and sometimes lab tests. Alvarez et al. [2] trained a logistic regression model using comprehensive EMR data to identify patients at high risk of out-of-ICU resuscitation events and death. They found fourteen important predictors, including age, oxygenation, diastolic blood pressure, arterial blood gas, lab test results, emergent orders, and assignment to a high-risk floor. Their automated model outperformed MEWS, obtaining a sensitivity of 42.2%, a specificity of 91.3%, and an advance warning of 15.9 hours. Their risk scores are not calculated in real-time; rather, data from the previous day are used to determine the score for the current day. In contrast, our formulation mimics the real-time streaming of EMR data into a prediction model.

In many cases, the patients’ conditions deteriorate quickly and suffers a cardiac arrest or death outside the ICU. Some studies consider out-of-ICU mortality as a secondary outcome and aim to find factors associated with it. Wengerter et al. [13] evaluated Rothman Index variability for predicting rapid response team activation as the primary outcome and in-hospital mortality as a secondary outcome. The Rothman Index incorporates twenty-six data points from EMR, including vital signs, lab test results, cardiac rhythm, and nursing assessments. It is based on the contribution of each factor to a model for predicting one-year mortality, but the Rothman Index itself is not modeled to predict any particular outcome. The authors considered the Rothman Index standard deviation (RISD) and the maximum-minus-minimum Rothman Index (MMRI) over a 24-hour window as two measures of variability. For predicting rapid response team activation, RISD and MMRI have AUCs of 0.74 and 0.76, sensitivities of 91.7% and 92.2%, and specificities of 39.9% and 37.3%, respectively. They found no association between these Rothman Index variability measures and in-hospital mortality.

Both MEWS and Rothman Index scores require manual entry by nurses and only consider small sets of data categories. Rothman Index only considers 26 data points and other work on predicting deterioration [20][21] uses smaller feature sets than ours and make use of 21 predictor variables. In contrast, we have upwards of over thousands of predictor variables across six different types of EMR data categories (shown in Table I). We believe our rich set of predictor variables not only improves the accuracy of our models but increases their robustness to missing data.

Another body of research targets ICU readmissions. ICU readmissions may be a more difficult target for machine learning than the first ICU transfer since the former involves patients who have been discharged from the ICU by a human expert (the physician) presumably after extensive tests and monitoring. Desautels et al. [4] learned a model to predict death and 48-hour ICU readmission when a patient is first discharged from the ICU. Their AdaBoost model containing 1,000 decision trees was trained on the MIMIC III dataset [7] and achieved an accuracy of 70%, a sensitivity of 59%, and a specificity of 66%.

In this paper, we built machine learning models for three hospitals to predict unplanned transfers of patients to the ICU as a proxy for clinical deterioration outside the ICU. The models are tunable with a good trade-off possibility between sensitivity and specificity. Also, the models take into account comprehensive data from EMR records and achieve AUCs above 0.85. When the specificity is set to 75%, the model sensitivities are around 80-85%, with greater advance warning compared to widely-used rule-based systems and other studies that apply predictive models.

With its higher sensitivity and earlier warnings, our approach has the potential to detect more patients with clinical deterioration as well as increase the chance of a successful intervention. The next steps are first to determine the performance of the model in a real-world clinical setting and then to see whether using the model in a Rapid Response System demonstrates improved outcomes.

In addition to learning a separate model for each hospital, we also tried learning a single model using the combined data from all three hospitals. The single-model results (not shown) are promising, with only a modest decrease in performance compared with the separate models. Having the same model work for multiple hospitals is important for both scalability and adoption, since it is impractical to create a separate model for each hospital that needs such a system—the cost of data collection and preparation alone will deter adoption.

For a solution provider, the biggest challenge to operationalizing a model is scalability. The data from different hospitals are heterogeneous and need to be normalized before they can be merged to create a single model. For example, the three hospitals in our study use different names to describe their wards, which we mapped to a common set of the most frequent names. To scale out our approach, significant work is needed to define common representations for all the important EMR data elements.

In general, value co-creation is an important aspect of innovation in the healthcare industry. Before developing any AI solution, machine learning scientists and healthcare administrators should have a clear value model and continue to collaborate after the integration of the solution in the hospital’s current IT system. In the case of our solution for predicting clinical deterioration, the biggest challenge for a hospital with an existing Rapid Response System is in transitioning from an intermittent system of patient monitoring to an always-on system using streaming data. Other issues that need to be addressed before operationalizing the solution are: 1) estimating the the efforts required to deal with hospital variations when we implement our system on a new hospital; 2) addressing model drift as data and clinical practice change over time; and 3) whether the value of the clinical improvements obtained by using the model outweigh the cost of implementing and maintaining the model.

In the healthcare analytics, there is a trade-off between a more generalized prediction model that inputs big data and global features and a more specific model with a targeted use case. The features that effectively characterize a condition are the same attributes that can train an accurate predictor. But if those features do not stand out above the background noise, then the predictor only finds the noise. For this reason, predictions that are focused on a specific clinical goal will always supersede a generic predictor in terms of accuracy and utility. Hence, the full power of clinical prediction is best realized when the computational question is carefully defined, specific variables are gathered, and a targeted need is met. Our study with the carefully defined goal and attentively characterized predictors that rooted in clinical knowledge of physicians is a good example to showcase the full power of clinical prediction in an important real-life application.