This paper was converted on www.awesomepapers.org from LaTeX by an anonymous user.
Want to know more? Visit the Converter page.

Explainable Machine Learning for ICU Readmission Prediction

Alex G. C. de Sá111These authors contributed equally to this work. Daniel Gould222These authors contributed equally to this work. Anna Fedyukova333These authors contributed equally to this work. Mitchell Nicholas Lucy Dockrell Calvin Fletcher David Pilcher Daniel Capurro David B. Ascher Khaled El-Khawas444These authors contributed equally to this work. Douglas E. V. Pires
Abstract

Background

The intensive care unit (ICU) comprises a complex hospital environment, where decisions made by clinicians have a high level of risk for the patients’ lives. Uncertain, competing and unplanned aspects within the ICU environment increase the difficulty in uniformly implementing the care pathway. Readmission contributes to this pathway’s difficulty, resulting in high mortality rates and resource utilisation.

Objectives

Several works have tried to predict readmission through patients’ medical information. Although they have some level of success while predicting readmission, those works do not properly assess, characterise and understand readmission prediction. This work proposes a standardised and explainable machine learning pipeline to model, predict and explain patient ICU readmission.

Method

This work focuses on using machine learning to model readmission using a multicentric eICU database while validating it on both MIMIC-IV monocentric and eICU multicentric settings. SHAP (Shapley Additive Explanations), calibration and likelihood ratio analysis are employed to comprehend the behaviour of the predictive model.

Results

Our machine learning pipeline achieved predictive performance in the area of the receiver operating characteristic curve (AUC) up to 0.7 with a Random Forest classification model, yielding an overall consistency and generalisation on validation sets. After an interpretation analysis of our model through likelihood ratios and calibration, we observed that the proposed methodology could generate predictive models with proper and translatable diagnosis capabilities. In addition, from explanations provided by the constructed model, we could also derive a set of insightful conclusions, primarily on variables related to vital signs and blood tests (e.g., albumin, blood urea nitrogen and haemoglobin levels), demographics (e.g., age, and admission height and weight), and ICU-associated variables (e.g., unit type).

Conclusions

The model’s predictions, behaviour and explainability method provided insights that yielded valuable information, which clinicians might (be able to) use to make decisions while discharging ICU patients.

keywords:
Readmission , Intensive Care Unit , Machine Learning , Explainable Predictions.
††journal: International Journal of Medical Informatics
\affiliation

[label1]organization=School of Chemistry and Molecular Biosciences, The University of Queensland, city=Brisbane City, state=Queensland, country=Australia \affiliation[label2]organization=Systems and Computational Biology, Bio21 Institute, The University of Melbourne, city=Parkville, state=Victoria, country=Australia

\affiliation

[label3]organization=Computational Biology and Clinical Informatics, Baker Heart and Diabetes Institute, city=Melbourne, state=Victoria, country=Australia

\affiliation

[label4]organization=Baker Department of Cardiometabolic Health, The University of Melbourne, city=Parkville, state=Victoria, country=Australia

\affiliation

[label7]organization=Faculty of Medicine, Dentistry and Health Sciences, The University of Melbourne, city=Parkville, state=Victoria, country=Australia

\affiliation

[label8]organization=The Department of Surgery, St. Vincent’s Hospital Melbourne, city=Fitzroy, state=Victoria, country=Australia

\affiliation

[label5]organization=Centre for the Digital Transformation of Health, School of Computing and Information Systems, The University of Melbourne, city=Parkville, state=Victoria, country=Australia

\affiliation

[label9]organization=The Department of Anaesthesia and Perioperative Medicine, Monash Health, city=Clayton, state=Victoria, country=Australia

\affiliation

[label10]organization=The Intensive Care Unit at Peninsula Private Hospital, Ramsay Healthcare, city=Langwarrin, state=Victoria, country=Australia

\affiliation

[label11]organization=The Department of Intensive Care Medicine, The Alfred Hospital, city= Prahran, state=Victoria, country=Australia

\affiliation

[label11b]organization=The Department of Anaesthesiology and Perioperative Medicine, The Alfred Hospital, city=Melbourne, state=Victoria, country=Australia

\affiliation

[label12]organization=The Australian and New Zealand Intensive Care Society Centre for Outcome and Resource Evaluation, city=Camberwell, state=Victoria, country=Australia

\affiliation

[label13]organization=The Australian and New Zealand Intensive Care Research Centre, School of Public Health and Preventive Medicine, Monash University, city=Melbourne, state=Victoria, country=Australia

\affiliation

[label14]organization=The Department of Intensive Care, Ballarat Base Hospital, city=Ballarat, state=Victoria, country=Australia

\affiliation

[label15]organization=The Department of Intensive Care, Austin Hospital, city=Heidelberg, state=Victoria, country=Australia

1 Introduction

Intensive care is a service for patients with likely recoverable and treatable conditions, providing them with more precise observation and invasive treatments [1]. Appropriate patient management within an intensive care unit (ICU) is crucial. In a given ICU, the likelihood of long-term length of stay (LOS), organ failures, and mortality tends to increase if adequate management is not taken into consideration [2, 3, 4, 5]. Clinicians who are caring for these patients also have multiple competing activities to consider [6, 7], including the potential deterioration of the patient’s condition after discharge, emergency admissions, elective admissions for high-risk surgery, staffing and resources, and evidence for appropriate allocation of resources. These aspects highlight the challenges across an ICU setting.

Readmission is one of the factors that extends the challenges within an ICU setting. It occurs when patients are admitted again to the ICU in a short timeframe (between 48 hours and 30 days), resulting in high mortality rates, increased LOS and, consequently, high resource utilisation [5, 8, 9, 10]. In summary, when readmission occurs, it disrupts the care pathway for the patients and poses additional challenges to the clinical team and hospital caring for them.

Reducing ICU readmission rates might not only improve the care pathway, leading to better patient outcomes, but also affect the hospital’s bottom line [11, 12]. Predicting patients at high risk of readmission would not only allow early intervention to reduce the risk but also reduce mortality rates, reduce resource utilisation (based on LOS) and, potentially, hospital costs.

The emergence of abundant hospital electronic health record (EHR) data [13, 14, 15] allowed the use of machine learning (ML) models targeting ICU readmission [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]. These models for general readmission prediction have been summarised in Table S1. Apart from their undeniable importance, these models lack proper interpretation, generalisation and validation in different ICU settings – e.g., monocentric (one hospital centre) versus multicentric (multiple ICU hospital centres).

Our main contention is that predicting ICU readmission in a well-generali-sable way with hospital-based datasets is still a challenge to overcome, even after taking into account these univariate predictive models and best practice ML techniques [22, 28, 29]. The main reason for this difficulty is that medical data is usually noisy, uncertain, and characterised by a high degree of missingness, as it relies on different human inputs. Further exacerbating this complexity is data imbalance, with only a small proportion of patients being readmitted. ICU patients who happen to be readmitted are also very heterogeneous [28], meaning that finding novel patterns across an ICU readmission data cohort tends to be challenging. As a result, an appropriate translation from ML models to the clinical environment is still limited. To improve the clinical translation, databases (e.g., MIMIC-IV and eICU) should be homogenised and have their noise and missingness diminished.

Accordingly, this work proposes a standardised methodological pipeline for a comprehensive assessment of ICU readmission modelling, prediction and understanding [30, 31] using machine learning models [32, 33] and statistical analyses. The produced predictive ICU readmission model – which was trained and internally validated in a dataset (i.e., eICU) containing several hospitals (multicentric) and externally validated in a monocentric dataset (i.e., MIMIC IV) [30, 31, 34] – shows clear signs of predictive generalisation. It is the first time in the literature that such an approach is employed to confirm the performance of an ICU readmission model. To the best of our knowledge, no other previous method has been validated in opposing ICU settings (i.e., monocentric and multicentric).

2 Material and Methods

Figure 1 presents the followed methodological workflow. The first step is to collect data from two databases: MIMIC IV (Medical Information Mart for Intensive Care, Version IV) [34, 35] and eICU (eICU Collaborative Research Database) [30, 31, 35]. From these databases, variables describing the patient’s information are created considering their admission timeframe (see Figure S1). These variables include patient demographics, pre-admission information, vital signs and blood test results, and also variables outlining the hospitals and ICUs. These variables are fully described in Tables S2 and S3.

Data preprocessing considers several steps, e.g., imputation, imbalance learning, and feature standardisation. Exclusion criteria are also taken into consideration to avoid abnormal patient information in our dataset (see Figure S2). Machine learning algorithms take these preprocessed features as input to build a readmission model based on a timeframe of 30 days. Hyperparameter optimisation also occurs (Table S4), guaranteeing the selection of the most appropriate predictive model for the data at hand. The final employed machine learning algorithms use multicentric data from eICU to build and internally validate the predictive model for readmission while externally testing and validating the final model on monocentric data from MIMIC-IV. The readmission model predicts whether a given patient will be readmitted or not and supports explanations for its predictions, driving insights to clinicians while discharging a given patient in the ICU.

Refer to caption
Figure 1: The methodological workflow followed by this work. Data comes from monocentric and multicentric databases (i.e., MIMIC IV and eICU, respectively), where the characterisation of ICU patients, hospitals, and ICU(s) takes place. Preprocessing filters out, standardises and imputes data for the development of a 30-day readmission machine learning model, which is built and validated on eICU and tested on MIMIC-IV. This model is interpreted and also used to drive explanations from variables and predictions, potentially assisting and guiding clinicians while treating and discharging new patients from the ICU.

The Supplementary Material (more specifically, Supplementary Methods) provides complete details about data sources (eICU, MIMIC IV, and readmission outcome), predictor variables, data preprocessing, machine learning modelling and validation, and the model’s explanation, diagnosis expression, and calibration.

3 Results

3.1 eICU Readmission Group Characteristics, Outcomes and Comparison to MIMIC IV

Table 1 provides the analyses contrasting readmitted and non-readmitted population groups. Out of 149,009 ICU admissions in the eICU database, readmission happens in 6,021 (4.04%) patients within 30 days of their first ICU admission. We have observed that the readmitted group includes older, sicker and more male patients. These results are corroborated by a statistical test with a significance level lower than 1%. The readmitted group also had a high level of gastrointestinal, respiratory and sepsis admission diagnoses when compared to the non-readmitted group (with a p-value << 1%).

The combined medical-surgical ICU and cardiothoracic surgical ICU types reached the lowest proportion of readmissions, with statistical significance. The readmitted group received more vasopressors and renal replacement therapies when contrasted with the non-readmitted group. The readmitted group commonly had a longer ICU stay and, consequently, a longer hospital length of stay (p-value << 1%). Finally, they were more than twice as likely to die in a hospital (16.0% versus 7.1% with a p-value << 1%), with survivors needing more rehabilitation, home nursing or skilled nursing facilities (p-value << 1%).

The Medical-Surgical Intensive Care Unit (MSICU) had the highest readmission rate at 49.1%, followed by the Medical Intensive Care Unit (MICU) with 10.9% and the Critical Care Unit-Cardio-Thoracic Intensive Care Unit (CCU-CTICU) with 10.1%. However, the absolute difference between readmission and non-readmission rates in most ICU types does not exceed 2.7%, except for MSICU, with a 6.6% difference between the two groups. The lowest readmission rate has been found in the Cardio-Surgical Intensive Care Unit (CSICU).

Table 1: Characteristics of discharged readmitted and non-readmitted patients in eICU.
Characteristic
Readmitted
(value with % OR
mean/median)
Non-Readmitted
(value with % OR
mean/median)
p-value
Readmission Numbers 6,021 142,988 -
Gender:
Male 3,417 (56.8%) 77,144 (54.0%) 2.198e-05*
Female 2,602 (43.2%) 65,771 (46.0%) 2.198e-05*
Age 65.48 / 67.00 62.70 / 65.00 <2.2e-16*
Body Mass Index (BMI) 28.69 / 27.30 29.05 / 27.51 <0.001*
APACHE IV 59.11 / 56.00 53.86 / 50.00 <2.2e-16*
APACHE Diagnosis:
Cardiovascular 1,789 (29.7%) 45,423 (31.8%) <0.001*
Neurological 881 (14.6%) 27,068 (18.9%) <2.2e-16*
Gastrointestinal 798 (13.3%) 13,755 (9.6%) <2.2e-16*
Trauma 230 (3.8%) 6,470 (4.5%) 0.011
Respiratory 1,045 (17.4%) 22,158 (15.5%) <0.001*
Sepsis 972 (16.1%) 17,404 (12.2%) <2.2e-16*
Other 412 (6.8%) 13,810 (9.7%) 3.8e-13*
Unit Type:
Critical Care Unit-Cardio-Thoracic Intensive Care Unit (CCU-CTICU) 610 (10.1%) 12,322 (8.6%) 4.8e-05*
Cardio-Surgical Intensive Care Unit (CSICU) 90 (1.5%) 5,335 (3.7%) <2.2e-16*
Cardio-Thoracic Intensive Care Unit (CTICU) 291 (4.8%) 4,750 (3.3%) 2.7e-10*
Cardiac Intensive Care Unit (CICU) 440 (7.3%) 9,392 (6.6%) 0.025
Medical Intensive Care Unit (MICU) 657 (10.9%) 11,710 (8.2%) 7.6e-14*
Surgical Intensive Care Unit (SICU) 503 (8.4%) 8,854 (6.2%) 1.5e-11*
Medical-Surgical Intensive Care Unit (MSICU) 2,955 (49.1%) 79,613 (55.7%) <2.2e-16*
Neurological Intensive Care Unit (NICU) 475 (7.89%) 11,012 (7.7%) 0.610
Hospital Capacity:
Extra-Large Hospital
(>500 beds)
2,427 (40.3%) 51,538 (36.0%) 1.7e-11*
Large Hospital
(250 – 499 beds)
1,431 (23.8%) 33,108 (23.2%) 0.277
Medium-sized Hospital
(100 – 249 beds)
1,052 (17.5%) 31,539 (22.1%) <2.2e-16*
Small hospital
(<100 beds)
231 (3.8%) 9,438 (6.6%) <2.2e-16*
Unknown number of beds 880 (14.6%) 17,365 (12.1%) 1.1e-08*
Hospital Type:
Teaching 1,687 (28.0%) 35,715 (25.0%) 1.1e-07*
Non-teaching 4,334 (72.0%) 107,273 (75.0%) 1.1e-07*
Patient Origin:
ICU 152 (2.5%) 3,261 (2.3%) 0.232
Operating Room 564 (9.4%) 14,675 (10.3%) 0.026
Emergency Department (ED) 2,089 (34.7%) 59,728 (41.8%) <2.2e-16*
Recovery Room 222 (3.7%) 4,929 (3.4%) 0.3359
Unknown 1,545 (25.7%) 33,334 (23.3%) 2.7e-05*
Other 1,449 (24.1%) 27,061 (18.9%) <2.2e-16*
Therapies in ICU:
Mechanical Ventilation 1,252 (20.8%) 27,651 (19.3%) 0.005
Vasopressors 761 (12.6%) 16,060 (11.2%) <0.001*
Renal Replacement Therapies (RRT) 232 (3.9%) 4,406 (03.1%) <0.001*
Length of Stay (LOS):
ICU LOS, days 3.72 / 2.18 2.86 / 1.75 <2.2e-16*
Hospital LOS, days 17.86 / 14.64 6.92 / 5.00 <2.2e-16*
Hospital Mortality 965 (16.0%) 10,162 (7.1%) <2.2e-16*
Discharge destination:
Home 2,290 (38.0%) 87,841 (61.4%) <2.2e-16*
Care Facility 2,409 (40.0%) 38,220 (26.7%) <2.2e-16*
Care Facility (Discharge destination):
Skilled Nursing Facility 1,150 (19.1%) 18,578 (13.0%) <2.2e-16*
Other Hospital 374 (6.2%) 5,698 (4.0%) <2.2e-16*
Rehabilitation 428 (7.1%) 6,213 (4.3%) <2.2e-16*
Nursing Home 128 (2.1%) 1,604 (1.1%) 1.7e-12*
Other External 329 (5.5%) 6,127 (4.3%) 1.2e-05*
⋆\star Presence of statistical significance.

Figure S3 also highlights patients readmitted to another type of ICU where the patient was originally admitted. Several ICUs in the eICU database have more than 30% readmissions to other ICU types, such as the Surgical Intensive Care Unit (SICU), Medical Intensive Care Unit (MICU) and Cardiac Intensive Care Unit (CICU).

While aiming to compare MIMIC IV and eICU readmitted patients, we noticed several differences in the level of information these cohorts provide, including patients who do not have length of stay (LOS) and discharge location. As a result, MIMIC IV and eICU were reduced from 5,984 and 6,021 to 5,980 and 5,403, respectively. This step was performed only to compare both cohorts directly. Matching the characteristics of MIMIC IV and eICU is not straightforward overall. MIMIC IV does not include several characteristics contained in eICU, and vice-versa, meaning that the comparison across the readmission groups from eICU and MIMIC IV was restricted to only a few characteristics.

Apart from that, we could observe they have similar baseline characteristics (Table 2). However, MIMIC IV readmitted patients received more ventilation and vasopressors but fewer renal replacement therapies than the readmitted eICU patients. In addition, MIMIC IV readmitted patients stayed longer in the ICU and hospital stay. Mortality was also slightly higher in the MIMIC IV readmission patients, although no statistical difference has been found between the two databases. Among survivors, the readmission group in eICU was discharged more to home, while readmitted patients in MIMIC IV had a higher rate of further treatments in a rehabilitation care facility (p-value << 1%).

Table 2: Comparisons of characteristics between eICU and MIMIC IV readmission groups.
Characteristic
eICU
(value with % OR
mean/median)
MIMIC
(value with % OR
mean/median)
p-value
Readmission Numbers 5,403 5,980 -
Gender:
Male 3,077 (56.9%) 3,382 (56.6%) 0.700
Female 2,326 (43.1%) 2,598 (43.4%) 0.685
Body Mass Index (BMI) 28.7 / 27.3 28.9 / 27.8 0.010
Age 65.5 / 67.0 65.7 / 67.0 0.620
Patient Origin:
Emergency Department (ED) 1,882 (34.8%) 2,824 (47.2%) <2.2e-16*
Therapies in ICU:
Mechanical Ventilation 1,146 (21.2%) 1,410 (23.6%) 0.003
Vasopressors 703 (13.0%) 1,111 (18.6%) 6.6e-16*
Renal Replacement Therapies (RRT) 215 (4.0%) 66 (1.1%) <2.2e-16*
Length of Stay (LOS):
ICU LOS, days 3.7 / 2.2 3.5 / 2.0 6.2e-05*
Hospital LOS, days 17.9 / 14.6 20.2 / 15.0 9.5e-06*
Hospital Mortality 882 (16.3%) 1,089 (18.2%) 0.009
Discharge destination:
Home 2,055 (38.0%) 631 (10.6%) <2.2e-16*
Care Facility (Rehabilitation) 407 (7.5%) 697 (11.7%) 1.5e-13*
Care Facility (Skilled Nursing Facility) 1,052 (19.5%) 1,132 (18.9%) 0.479
⋆\star Presence of statistical significance.

3.2 Machine Learning Validation

The statistical analysis provided an insightful characterisation of the readmission group, contrasting it with the non-readmission group and across populations. With this information, we can now move to a machine learning analysis and validation.

Table S5 shows the 47 selected features by the greedy feature selection approach using a Random Forest classifier and considering a balanced readmission dataset. The details of these variables representing machine learning features can be found in Table S2. Although we employed several different undersampling techniques to tackle the high level of readmission imbalance, using a balanced readmission rate presented itself as the most successful technique, which was then applied in both eICU and MIMIC IV cohorts at first. Hyper-parameter optimisation was also applied after feature selection, with the aim of improving the predictive performances of the resultant models (Table S4).

When bringing all these ML components together, our proposed pipeline achieved an area under the ROC curve (AUC) of 0.68 on multicentric eICU data under a 10-fold cross-validation procedure with a Random Forest classifier with 80 trees in a balanced readmission scenario. It is worth noting that this classification model was considered the best against 9680 other machine learning models in terms of a trade-off among AUC, MCC and recall metrics on 10-fold cross-validation. Explainability was also a key point in selecting the model. Therefore, the selected variables (features) were taken into consideration while selecting the predictive model itself. On the blind test (stratified 10% from eICU data), which internally validates the proposed model, consistent results were reached by the predictive model (AUC of 0.672) when compared to 10-fold cross-validation. When externally validated on MIMIC IV’s data, our model reached an AUC of 0.616, which demonstrates the overall generalisation capabilities of the proposed model to predict 30-day readmission. Figure 2 shows the AUC plots for these three validation schemes. The results for other performance measures are also summarised in Table S6.

Refer to caption
Figure 2: The predictive performance of the proposed readmission model on 10-fold cross-validation and blind test on eICU data. External validation was made utilising MIMIC IV data.

3.3 Calibration and Likelihood Analysis

Based on the performed Calibration Analysis (Supplementary Results, Table S7 and Figures S4-S6), the calibration curves, defined from our 30-day ICU readmission model during cross-validation, blind testing and external testing, indicated slight overestimation/underestimation. Therefore, our main conclusion is that our model is well-calibrated in all sets, demonstrating its clinical usability in an ICU setting.

3.4 Likelihood Ratio Analysis

In addition, our 30-day ICU readmission model faced an analysis based on Likelihood Ratios (see Supplementary Results and Figure S7), which highlighted the importance of properly setting the classification probability threshold. This analysis step is beneficial for having a better diagnostic readmission profile for ICU patients. In summary, the ICU predictive model’s diagnostic power depends on a proper analysis of these thresholds, which was properly defined after the Likelihood Ratio Analysis (Supplementary Results).

3.5 Explainable Machine Learning for ICU Readmission

We used SHAP to demonstrate how individual features included in the model (see Tables S2 and S5) influence overall readmission predictions. Figure 3 summarises and highlights the 20 most important features, based on SHAP values, we employed to characterise ICU readmission in eICU and MIMIC IV.

Refer to caption
Figure 3: The SHAP summary plot for our proposed readmission model on eICU training data. We show the 20 features with higher SHAP values, i.e., that have a higher impact on the model’s predictive outputs.

Most of the features in the SHAP plot of Figure 3 are vital sign- and blood test-related (see Table S2). The three most important in this category are the minimum value of albumin in the blood during the first 24 hours, the maximum level of blood urea nitrogen (BUN) during the last 24 hours and the minimum level of hemoglobin during the last 24 hours.

We identified through the SHAP tree explanation model that high values for the variables Min. Albumin (First 24hs) and Min. Hemoglobin (Last 24hs) are usually more linked to the non-readmission of patients, while low values for them are more present in readmitted patients. Max. Blood Urea Nitrogen (Last 24hs) reveals an opposite trend, where higher values for this variable are fairly more associated with ICU patient readmission, and there is clear evidence that low values for the Max. Blood Urea Nitrogen (Last 24hs) result are attached to non-readmission.

4 Discussion

Although it is challenging to map and match common variables or characteristics across different cohorts, our study drove important clinical insights while analysing eICU and comparing it to MIMIC IV (Section 3.1). We discovered how heterogeneous the readmission monocentric and multicentric populations are in general. For example, the readmitted patients in MIMIC IV were treated more frequently with ventilation and vasopressors, although renal replacement therapies were more common in patients readmitted to eICU. Such contrasts in the cohort populations highlight the ability of our standardised pipeline to deal with heterogeneous readmission data.

Our proposed machine learning pipeline learned proper patterns on multicentric ICU data, consequently generalising well on an independent blind test set over the same data (Section 3.2). Our proposed machine learning model captured the essence of readmission, being able to transfer similar predictive performance to external validation on monocentric ICU data (Section 3.2). Overall, this shows that learning from data coming from multiple ICUs has its limitations but may lead to generalisable predictive performance if done adequately.

A reasonable calibration level is also an essential aspect of the proposed readmission model (Supplementary Results). As a result, our proposed model shares meaningful clinical decision-making due to its ability to calculate individualised probabilistic estimates of readmission [36]. For example, clinicians may decide that patients with a risk of readmission at least double that of the average risk in their ICU population should be flagged for routine monitoring following discharge from ICU. If the patient is moved to the ward, this may take the form of a brief clinical review by an intensivist each day following transfer to identify early signs of clinical deterioration. This could potentially facilitate the escalation of ward-based measures to optimise the patient’s management and avoid ICU readmission, or it could facilitate better preparation and planning such that unplanned readmissions are minimised.

A good diagnostic level, when we increase the classification probability threshold, complements the model’s predictive performance (Supplementary Results). At the threshold(s) of 0.712 (and 0.813), our model returns the most reliable likelihood ratio among cross-validation, blind testing and external validation, yielding trustworthy readmission predictions.

Finally, we highlight how the 20 most impactful features are linked with readmission prediction through their explainability (Section 3.3 and Supplementary Results) [17, 37, 38, 39, 21, 40, 41, 42, 43, 44, 45, 46, 47]. Overall, this shows our proposed model was able to learn and capture good insights on eICU data, being compatible with results and discussion in the ICU readmission literature. Supplementary Discussion provides more details on the model’s explainability, strength and limitations of this work and its relationships to other studies.

Authors’ contributions

Alex G. C. de Sá: Conceptualisation; Formal analysis; Investigation; Methodology; Project administration; Resources; Software; Validation; Visualisation; Writing – original draft; Writing – review & editing.

Daniel Gould: Conceptualisation; Formal analysis; Investigation; Methodology; Project administration; Validation; Visualisation; Writing – original draft; Writing – review & editing.

Anna Fedyukova: Data curation; Formal analysis; Methodology; Software; Validation Visualisation; Writing – original draft.

Mitchell Nicholas: Validation; Writing – review & editing.

Calvin Fletcher: Validation; Writing – original draft; Writing – review & editing.

Lucy Dockrell: Validation; Writing – review & editing.

David Pilcher: Supervision; Validation; Writing – review & editing.

Daniel Capurro: Conceptualisation; Validation; Writing – review & editing.

David B. Ascher: Supervision; Validation; Writing – review & editing.

Khaled El-Khawas: Conceptualisation; Investigation; Methodology; Project administration; Supervision; Validation; Writing – original draft; Writing – review & editing.

Douglas E. V. Pires: Conceptualisation; Methodology; Project administration; Supervision Validation; Writing – original draft; Writing – review & editing.

Acknowledgements

D.B. A. was supported by an Investigator Grant from the National Health and Medical Research Council (NHMRC) of Australia (GNT1174405 to D.A.). Supported in part by the Victorian Government’s Operational Infrastructure Support Program.

Statement on Conflicts of Interest

None declared.

Summary Table

  • 1.

    We propose a standardised and explainable machine learning pipeline for 30-day readmission prediction in multicentric eICU and monocentric MIMIC IV cohorts.

  • 2.

    Our random forest-based readmission model yields overall good calibration and generalisable predictive performance on internal and external validation sets.

  • 3.

    Calibration and likelihood ratio analysis also demonstrated the proper clinical (diagnostic) usability of our proposed model, even with the challenging nature of the studied readmission datasets.

  • 4.

    The derived explanations from the proposed readmission model provide invaluable guidelines, which might be helpful for ICU clinicians’ decision-making while discharging patients.

Supplementary Information

Supplementary Materials are available at https://tinyurl.com/fne6xv2t.

References

  • Smith and Nielsen [1999] G. Smith, M. Nielsen, Criteria for admission, The British Medical Journal (BMJ) 318 (1999) 1544–1547. URL: https://doi.org/10.1136/bmj.318.7197.1544.
  • Moitra et al. [2016] V. Moitra, C. Guerra, W. Linde-Zwirble, H. Wunsch, Relationship between ICU length of stay and long-term mortality for elderly ICU survivors, Critical Care Medicine 44 (2016) 655. URL: https://doi.org/10.1097%2FCCM.0000000000001480.
  • Sakr et al. [2012] Y. Sakr, S. Lobo, R. Moreno, H. Gerlach, M. Ranieri, A. Michalopoulos, J.-L. Vincent, Patterns and early evolution of organ failure in the intensive care unit and their relation to outcome, Critical Care 16 (2012) 1–9. URL: https://doi.org/10.1186/cc11868.
  • Ay et al. [2020] E. Ay, M. Weigand, R. Röhrig, M. Gruss, et al., Dying in the intensive care unit (ICU): A retrospective descriptive analysis of deaths in the ICU in a communal tertiary hospital in germany, Anesthesiology Research and Practice 2020 (2020). URL: https://doi.org/10.1155/2020/2356019.
  • Zhang and Kuo [2024] M. Zhang, T.-T. Kuo, Early prediction of long hospital stay for intensive care units readmission patients using medication information, Computers in Biology and Medicine 174 (2024) 108451.
  • Sujan et al. [2015] M. Sujan, P. Chessum, M. Rudd, L. Fitton, M. Inada-Kim, M. Cooke, P. Spurgeon, Managing competing organizational priorities in clinical handover across organizational boundaries, Journal of Health Services Research & Policy 20 (2015) 17–25. URL: https://doi.org/10.1177/1355819614560449.
  • Turnbull et al. [2018] A. Turnbull, S. Sahetya, L. D. Biddison, C. Hartog, G. Rubenfeld, D. Benoit, B. Guidet, R. Gerritsen, M. Tonelli, R. Curtis, Competing and conflicting interests in the care of critically ill patients, Intensive Care mMdicine 44 (2018) 1628–1637. URL: https://doi.org/10.1007/s00134-018-5326-2.
  • Mcneill et al. [2020] H. Mcneill, S. Khairat, et al., Impact of intensive care unit readmissions on patient outcomes and the evaluation of the national early warning score to prevent readmissions: literature review, JMIR Perioperative Medicine 3 (2020) e13782. URL: https://doi.org/10.2196%2F13782.
  • Alban et al. [2006] R. Alban, A. Nisim, J. Ho, G. Nishi, M. M. Shabot, Readmission to surgical intensive care increases severity-adjusted patient mortality, Journal of Trauma and Acute Care Surgery 60 (2006) 1027–1031. URL: https://doi.org/10.1097/01.ta.0000218217.42861.b7.
  • Kramer et al. [2013] A. Kramer, T. Higgins, J. Zimmerman, The association between ICU readmission rate and patient outcomes, Critical Care Medicine 41 (2013) 24–33. URL: https://doi.org/10.1097/ccm.0b013e3182657b8a.
  • James [2013] J. James, Medicare hospital readmissions reduction program, Health Affairs 34 (2013) 1–5. URL: https://doi.org/10.1377/hpb20131112.646839.
  • Khera and Krumholz [2018] R. Khera, H. Krumholz, Effects of the hospital readmissions reduction program: The MedPAC report, Circulation: Cardiovascular Quality and Outcomes (2018). URL: https://doi.org/10.1161/CIRCOUTCOMES.118.005083.
  • Yadav et al. [2018] P. Yadav, M. Steinbach, V. Kumar, G. Simon, Mining electronic health records (EHRs): A survey, ACM Computing Surveys (CSUR) 50 (2018) 1–40. URL: https://doi.org/10.1145/3127881.
  • Kim et al. [2019] E. Kim, S. Rubinstein, K. Nead, A. Wojcieszynski, P. Gabriel, J. Warner, The evolving use of electronic health records (EHR) for research, Seminars in Radiation Oncology 29 (2019) 354–361. URL: https://doi.org/10.1016/j.semradonc.2019.05.010.
  • Shickel et al. [2017] B. Shickel, P. Tighe, A. Bihorac, P. Rashidi, Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis, IEEE Journal of Biomedical and Health Informatics 22 (2017) 1589–1604. URL: https://doi.org/10.1109/jbhi.2017.2767063.
  • Badawi and Breslow [2012] O. Badawi, M. Breslow, Readmissions and death after ICU discharge: development and validation of two predictive models, PloS One 7 (2012) e48758. URL: https://doi.org/10.1371/journal.pone.0048758.
  • Rojas et al. [2018] J. Rojas, K. Carey, D. Edelson, L. Venable, M. Howell, M. Churpek, Predicting intensive care unit readmission with machine learning using electronic health record data, Annals of the American Thoracic Society 15 (2018) 846–853. URL: https://doi.org/10.1513/annalsats.201710-787oc.
  • Mohanty et al. [2022] S. Mohanty, D. Lekan, T. McCoy, M. Jenkins, P. Manda, Machine learning for predicting readmission risk among the frail: Explainable AI for healthcare, Patterns 3 (2022). URL: https://doi.org/10.1016/j.patter.2021.100395.
  • Thoral et al. [2021] P. Thoral, M. Fornasa, D. de Bruin, M. Tonutti, H. Hovenkamp, R. Driessen, A. Girbes, M. Hoogendoorn, P. Elbers, Explainable machine learning on AmsterdamUMCdb for ICU discharge decision support: uniting intensivists and data scientists, Critical Care Explorations 3 (2021). URL: https://doi.org/10.1097/cce.0000000000000529.
  • McWilliams et al. [2019] C. McWilliams, D. Lawson, R. Santos-Rodriguez, I. Gilchrist, A. Champneys, T. Gould, M. Thomas, C. Bourdeaux, Towards a decision support tool for intensive care discharge: machine learning algorithm development using electronic healthcare data from MIMIC-III and Bristol, UK, BMJ Open 9 (2019) e025925. URL: https://doi.org/10.1136/bmjopen-2018-025925.
  • Hammer et al. [2021] M. Hammer, S. Grabitz, B. Teja, K. Wongtangman, M. Serrano, S. Neves, S. Siddiqui, X. Xu, M. Eikermann, A tool to predict readmission to the intensive care unit in surgical critical care patients—the RISC score, Journal of Intensive Care Medicine 36 (2021) 1296–1304. URL: https://doi.org/10.1177/0885066620949164.
  • Lin et al. [2019] Y.-W. Lin, Y. Zhou, F. Faghri, M. Shaw, R. Campbell, Analysis and prediction of unplanned intensive care unit readmission using recurrent neural networks with long short-term memory, PloS One 14 (2019) e0218942. URL: https://doi.org/10.1371/journal.pone.0218942.
  • Xue et al. [2019] Y. Xue, D. Klabjan, Y. Luo, Predicting ICU readmission using grouped physiological and medication trends, Artificial Intelligence in Medicine 95 (2019) 27–37. URL: https://doi.org/10.1016/j.artmed.2018.08.004.
  • Barbieri et al. [2020] S. Barbieri, J. Kemp, O. Perez-Concha, S. Kotwal, M. Gallagher, A. Ritchie, L. Jorm, Benchmarking deep learning architectures for predicting readmission to the ICU and describing patients-at-risk, Scientific Reports 10 (2020) 1111. URL: https://doi.org/10.1038/s41598-020-58053-z.
  • Hegselmann et al. [2022] S. Hegselmann, C. Ertmer, T. Volkert, A. Gottschalk, M. Dugas, J. Varghese, Development and validation of an interpretable 3 day intensive care unit readmission prediction model using explainable boosting machines, Frontiers in Medicine 9 (2022) 960296. URL: https://doi.org/10.3389/fmed.2022.960296.
  • Ashfaq et al. [2019] A. Ashfaq, A. Sant’Anna, M. Lingman, S. Nowaczyk, Readmission prediction using deep learning on electronic health records, Journal of Biomedical Informatics 97 (2019) 103256. URL: https://doi.org/10.1016/j.jbi.2019.103256.
  • Lv et al. [2023] J. Lv, M. Zhang, Y. Fu, M. Chen, B. Chen, Z. Xu, X. Yan, S. Hu, N. Zhao, An interpretable machine learning approach for predicting 30-day readmission after stroke, International Journal of Medical Informatics 174 (2023) 105050.
  • Hosein et al. [2013] S. Hosein, N. Bobrovitz, S. Berthelot, D. Zygun, W. Ghali, H. Stelfox, A systematic review of tools for predicting severe adverse events following patient discharge from intensive care units, Critical Care 17 (2013) 1–10. URL: https://doi.org/10.1186/cc12747.
  • Curto et al. [2016] S. Curto, J. Carvalho, C. Salgado, S. Vieira, J. Sousa, Predicting ICU readmissions based on bedside medical text notes, in: Proceedings of the IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), IEEE, 2016, pp. 2144–a–2151–h. URL: https://doi.org/10.1109/FUZZ-IEEE.2016.7737956.
  • Pollard et al. [2018] T. Pollard, A. Johnson, J. Raffa, L. Celi, R. Mark, O. Badawi, The eICU Collaborative Research Database, a freely available multi-center database for critical care research, Scientific Data 5 (2018) 1–13. URL: https://doi.org/10.1038/sdata.2018.178.
  • Pollard et al. [2019] T. Pollard, A. Johnson, J. Raffa, L. Celi, O. Badawi, R. Mark, eICU Collaborative Research Database (version 2.0), PhysioNet (2019). URL: https://doi.org/10.13026/C2WM1R.
  • Mitchell [1997] T. Mitchell, Machine Learning, 1 ed., McGraw-Hill, Inc., 1997. URL: https://dl.acm.org/doi/10.5555/541177.
  • Raschka et al. [2022] S. Raschka, Y. Liu, V. Mirjalili, Machine Learning with PyTorch and Scikit-Learn, 1st ed., Packt Publishing Ltd., 2022.
  • Johnson et al. [2021] A. Johnson, L. Bulgarelli, T. Pollard, S. Horng, L. A. Celi, R. Mark, MIMIC-IV, PhysioNet (2021). URL: https://doi.org/10.13026/s6n6-xd98.
  • Goldberger et al. [2000] A. Goldberger, L. Amaral, L. Glass, J. Hausdorff, P. Ivanov, R. Mark, J. Mietus, G. Moody, C.-K. Peng, E. Stanley, PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals, Circulation 101 (2000) e215–e220. URL: https://doi.org/10.1161/01.cir.101.23.e215.
  • Munn et al. [2022] J. Munn, B. Lanting, S. MacDonald, L. Somerville, J. Marsh, D. Bryant, B. Chesworth, Logistic regression and machine learning models cannot discriminate between satisfied and dissatisfied total knee arthroplasty patients, The Journal of Arthroplasty 37 (2022) 267–273. URL: https://doi.org/10.1016/j.arth.2021.10.017.
  • Herrmann et al. [1992] F. Herrmann, C. Safran, S. Levkoff, K. Minaker, Serum albumin level on admission as a predictor of death, length of stay, and readmission, Archives of Internal Medicine 152 (1992) 125–130. URL: https://doi.org/10.1001/archinte.1992.00400130135017.
  • Arihan et al. [2018] O. Arihan, B. Wernly, M. Lichtenauer, M. Franz, B. Kabisch, J. Muessig, M. Masyuk, A. Lauten, P. C. Schulze, U. Hoppe, et al., Blood Urea Nitrogen (BUN) is independently associated with mortality in critically ill patients admitted to ICU, PloS One 13 (2018) e0191697. URL: https://doi.org/10.1371/journal.pone.0191697.
  • Li et al. [2021] X. Li, R. Zheng, T. Zhang, Z. Zeng, H. Li, J. Liu, Association between blood urea nitrogen and 30-day mortality in patients with sepsis: a retrospective analysis, Annals of Palliative Medicine 10 (2021) 11653–11663. URL: https://doi.org/10.21037/apm-21-2937.
  • Warner et al. [2022] M. Warner, A. Hanson, P. Schulte, N. Roubinian, C. Storlie, G. Demuth, O. Gajic, D. Kor, Early post-hospitalization hemoglobin recovery and clinical outcomes in survivors of critical illness: a population-based cohort study, Journal of Intensive Care Medicine 37 (2022) 1067–1074. URL: https://doi.org/10.1177/08850666211069098.
  • Kerfeld et al. [2020] M. Kerfeld, D. Kor, R. Frank, A. Hanson, M. Passe, M. Warner, Hospital discharge hemoglobin values and posthospitalization clinical outcomes in transfused patients undergoing noncardiac surgery, Transfusion 60 (2020) 2250–2259. URL: https://doi.org/10.1111%2Ftrf.16002.
  • Lee et al. [2022] S.-I. Lee, Y. Koh, J. W. Huh, S.-B. Hong, C.-M. Lim, Factors and outcomes of intensive care unit readmission in elderly patients, Gerontology 68 (2022) 280–288. URL: https://doi.org/10.1159/000516297.
  • Glans et al. [2020] M. Glans, A. Kragh Ekstam, U. Jakobsson, Å. Bondesson, P. Midlöv, Risk factors for hospital readmission in older adults within 30 days of discharge–a comparative retrospective study, BMC Geriatrics 20 (2020) 1–12. URL: https://doi.org/10.1186/s12877-020-01867-3.
  • Pedersen et al. [2017] M. K. Pedersen, G. Meyer, L. Uhrenfeldt, Risk factors for acute care hospital readmission in older persons in western countries: a systematic review, JBI Evidence Synthesis 15 (2017) 454–485. URL: https://doi.org/10.11124/jbisrir-2016-003267.
  • Howie-Esquivel et al. [2019] J. Howie-Esquivel, K. Dracup, M. Whooley, C. McCulloch, C. Jin, D. Moser, R. Clark, M. Pelter, M. Biddle, L. Park, Rapid 5 lb weight gain is not associated with readmission in patients with heart failure, ESC Heart Failure 6 (2019) 131–137. URL: https://doi.org/10.1002/ehf2.12370.
  • You et al. [2013] J.-W. You, S. Lee, Y. Kim, Y. Cho, Y. Jeong, H. Kim, J. Lee, J. Kim, Y. Hwang, Association between weight change and clinical outcomes in critically ill patients, Journal of Critical Care 28 (2013) 923–927. URL: https://doi.org/10.1016/j.jcrc.2013.07.055.
  • Mishra et al. [2021] R. Mishra, A. Pande, R. Ramachandran, A. Trikha, P. Singh, V. Rewari, Effect of change in body weight on clinical outcomes in critically ill patients, Indian Journal of Critical Care Medicine 25 (2021) 1042. URL: https://doi.org/10.5005/jp-journals-10071-23978.